Sep 19 10:34:08.573034 ip-10-0-129-220 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 19 10:34:08.573046 ip-10-0-129-220 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 19 10:34:08.573053 ip-10-0-129-220 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 19 10:34:08.573205 ip-10-0-129-220 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 19 10:34:18.647719 ip-10-0-129-220 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 19 10:34:18.647738 ip-10-0-129-220 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b6d047af7b6d4f808340ff4a9c21b297 --
Sep 19 10:35:54.044224 ip-10-0-129-220 systemd[1]: Starting Kubernetes Kubelet...
Sep 19 10:35:54.465946 ip-10-0-129-220 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 10:35:54.465946 ip-10-0-129-220 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 19 10:35:54.465946 ip-10-0-129-220 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 10:35:54.465946 ip-10-0-129-220 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 19 10:35:54.465946 ip-10-0-129-220 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 10:35:54.466888 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.466796    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 19 10:35:54.475250 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475221    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 10:35:54.475250 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475245    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 10:35:54.475250 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475249    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 10:35:54.475250 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475252    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 10:35:54.475250 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475257    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 10:35:54.475250 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475264    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475267    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475270    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475273    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475276    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475279    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475282    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475285    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475288    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475291    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475293    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475296    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475299    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475304    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475307    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475313    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475316    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475319    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475322    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475324    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 10:35:54.475510 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475327    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475330    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475332    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475335    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475337    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475340    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475345    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475348    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475351    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475353    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475357    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475362    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475364    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475367    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475371    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475374    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475376    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475379    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475385    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 10:35:54.476008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475387    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475390    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475392    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475395    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475398    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475400    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475403    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475406    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475408    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475411    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475414    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475419    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475422    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475425    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475427    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475430    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475432    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475435    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475437    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475440    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 10:35:54.476497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475443    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475446    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475449    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475452    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475457    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475459    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475463    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475466    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475469    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475471    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475474    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475477    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475480    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475482    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475485    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475488    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475495    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475507    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475513    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475517    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 10:35:54.476993 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475520    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475523    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475526    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475529    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475532    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475535    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475539    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475545    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475548    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475550    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475553    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475556    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475559    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475562    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.475565    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477055    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477063    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477066    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477069    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477072    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 10:35:54.478262 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477074    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477077    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477080    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477083    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477086    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477091    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477094    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477096    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477099    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477102    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477105    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477108    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477111    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477113    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477118    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477122    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477125    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477128    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477134    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477137    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 10:35:54.478782 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477139    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477142    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477144    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477147    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477151    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477154    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477156    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477160    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477163    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477165    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477170    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477173    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477176    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477178    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477181    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477184    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477186    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477189    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477192    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477194    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 10:35:54.479310 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477197    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477200    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477202    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477207    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477210    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477213    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477216    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477220    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477224    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477227    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477230    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477233    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477238    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477242    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477244    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477250    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477253    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477256    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477259    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 10:35:54.479821 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477262    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477265    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477268    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477270    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477273    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477276    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477279    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477282    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477287    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477290    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477293    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477296    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477299    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477302    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477305    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477307    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477310    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477312    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477315    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477318    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 10:35:54.480351 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477324    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477327    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477330    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477333    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477335    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477338    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477340    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477343    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477345    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477348    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477351    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477354    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477359    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477362    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.477364    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478087    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478094    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478097    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478099    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478102    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 10:35:54.480835 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478105    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478107    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478110    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478112    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478115    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478117    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478120    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478122    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478125    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478127    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478129    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478132    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478134    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478137    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478139    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478142    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478144    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478147    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478149    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478152    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 10:35:54.481334 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478154    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478157    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478160    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478162    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478165    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478167    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478169    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478172    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478175    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478177    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478180    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478182    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478185    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478187    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478190    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478192    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478195    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478197    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478199    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478202    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 10:35:54.481824 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478204    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478207    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478210    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478212    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478215    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478220    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478223    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478225    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478227    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478230    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478232    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478235    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478238    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478240    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478242    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478245    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478248    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478250    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478252    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478255    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 10:35:54.482314 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478257    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478260    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478269    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478272    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478275    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478278    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478280    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478283    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478286    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478288    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478291    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478293    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478296    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478300    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478303    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478305    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478308    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478312    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478314    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478317    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 10:35:54.482792 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478319    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478323    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478327    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478331    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478335    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478338    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478343    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478346    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478348    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478351    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478354    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478356    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478359    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.478362    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478446    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478481    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478488    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478492    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478497    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478500    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 19 10:35:54.483298 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478504    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478508    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478511    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478514    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478517    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478522    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478525    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478528    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478531    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478534    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478538    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478540    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478545    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478547    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478550    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478554    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478557    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478561    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478564    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478567    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478570    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478573    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478576    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478579    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 19 10:35:54.483786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478582    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478585    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478589    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478592    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478594    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478597    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478600    2580 flags.go:64] FLAG: --enable-server="true"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478603    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478606    2580 flags.go:64] FLAG: --event-burst="100"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478609    2580 flags.go:64] FLAG: --event-qps="50"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478612    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478616    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478618    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478623    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478626    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478629    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478632    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478635    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478638    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478641    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478644    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478647    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478650    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478653    2580 flags.go:64] FLAG: --feature-gates=""
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478656    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 19 10:35:54.484409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478659    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478662    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478665    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478668    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478671    2580 flags.go:64] FLAG: --help="false"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478674    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478677    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478680    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478683    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478686    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478690    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478693    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478696    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478698    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478701    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478704    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478707    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478710    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478712    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478715    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478721    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478724    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478727    2580 flags.go:64] FLAG: --lock-file=""
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478729    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 19 10:35:54.485089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478732    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478735    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478741    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478744    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478747    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478750    2580 flags.go:64] FLAG: --logging-format="text"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478753    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478757    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478759    2580 flags.go:64] FLAG: --manifest-url=""
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478762    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478771    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478774    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478778    2580 flags.go:64] FLAG: --max-pods="110"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478781    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478784    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478787    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478790    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478793    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478796    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478799    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478806    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478809    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478812    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 19 10:35:54.485679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478815    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478818    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478826    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478829    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478831    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478834    2580 flags.go:64] FLAG: --port="10250"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478838    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478841    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-083e5ff1e7059a091"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478844    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478847    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478850    2580 flags.go:64] FLAG: --register-node="true"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478853    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478857    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478860    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478863    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478866    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478869    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478872    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478875    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478878    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478880    2580 flags.go:64] FLAG: --runonce="false"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478883    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478886    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478889    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478892    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 19 10:35:54.486325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478894    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478897    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478917    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478921    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478926    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478929    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478932    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478934    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478937    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478941    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478944    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478950    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478953    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478957    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478961    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478964    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478966    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478969    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478972    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478975    2580 flags.go:64] FLAG: --v="2"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478979    2580 flags.go:64] FLAG: --version="false"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478984    2580 flags.go:64] FLAG: --vmodule=""
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478988    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.478991    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479102    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 10:35:54.487057 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479106    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479109    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479112    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479115    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479118    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479120    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479123    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479125    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479128    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479130    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479133    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479135    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479138    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479140    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479143    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479145    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479149    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479152    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479154    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479157    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 10:35:54.487691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479161    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479165    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479168    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479172    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479175    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479178    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479181    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479184    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479186    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479189    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479192    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479194    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479197    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479199    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479202    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479204    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479207    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479209    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479211    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 10:35:54.488215 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479214    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479217    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479219    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479222    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479224    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479227    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479229    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479232    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479234    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479238    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479240    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479243    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479247    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479252    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479255    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479257    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479260    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479263    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479265    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 10:35:54.488686 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479268    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479270    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479273    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479275    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479278    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479281    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479283    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479286    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479288    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479291    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479293    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479295    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479298    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479301    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479303    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479305    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479308    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479310    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479313    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479316    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 10:35:54.489173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479318    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479321    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479325    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479327    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479330    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479332    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479336    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479339    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479341    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479344    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479346    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479349    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479351    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479354    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479356    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479359    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479361    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479366    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479369    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 10:35:54.489671 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.479372    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.480050    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.488324    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.488342    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.488357    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.488363    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488427    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488432    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488436    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488439    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488441    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488444    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488447    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488450    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488452    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 10:35:54.490252 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488456    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488459    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488461    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488464    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488467    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488469    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488472    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488474    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488477    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488479    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488482    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488484    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488487    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488489    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488492    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488494    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488497    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488499    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488502    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488504    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 10:35:54.490633 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488509    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488513    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488516    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488519    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488522    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488525    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488527    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488530    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488533    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488535    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488538    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488541    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488543    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488545    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488548    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488550    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488553    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488556    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488559    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 10:35:54.491148 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488561    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488564    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488567    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488569    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488572    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488574    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488577    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488579    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488582    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488584    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488587    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488590    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488592    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488595    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488598    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488601    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488604    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488607    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488609    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488612    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 10:35:54.491620 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488615    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488618    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488620    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488624    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488627    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488631    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488633    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488636    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488639    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488642    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488645    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488648    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488651    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488654    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488656    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488659    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488662    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488664    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488667    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 10:35:54.492136 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488669    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488672    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488675    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488679    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488682    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488684    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488688    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488690    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488693    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488696    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488699    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488701    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.488707    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488850    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 10:35:54.492600 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488855    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488858    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488861    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488864    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488866    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488869    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488872    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488875    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488877    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488880    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488884    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488888    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488891    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488894    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488897    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488913    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488916    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488919    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488921    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 10:35:54.493033 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488924    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488926    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488929    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488931    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488934    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488937    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488939    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488942    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488948    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488951    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488954    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488956    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488959    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488961    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488964    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488966    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488969    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488971    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488974    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488976    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 10:35:54.493489 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488979    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488981    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488984    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488986    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488989    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488991    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488995    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.488998    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489001    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489004    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489007    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489010    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489012    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489015    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489017    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489020    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489022    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489025    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489027    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489030    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 10:35:54.493987 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489033    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489036    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489038    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489041    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489043    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489046    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489048    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489050    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489053    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489055    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489058    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489060    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489063    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489065    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489068    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489070    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489074    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489077    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489080    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489083    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 10:35:54.494499 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489086    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489088    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489091    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489093    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489096    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489098    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489100    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489103    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489105    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489108    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489111    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489113    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489116    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489118    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489122    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489124    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489127    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489129    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 10:35:54.495008 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:54.489131    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 10:35:54.495442 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.489136    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 10:35:54.495442 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.489255    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 19 10:35:54.495442 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.493715    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 19 10:35:54.495442 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.494822    2580 server.go:1019] "Starting client certificate rotation"
Sep 19 10:35:54.495442 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.494930    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 10:35:54.495765 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.495752    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 10:35:54.497672 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.497630    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 10:35:54.499269 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.499246    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 10:35:54.509539 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.509522    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 19 10:35:54.511649 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.511633    2580 log.go:25] "Validated CRI v1 image API"
Sep 19 10:35:54.514314 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.514294    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 19 10:35:54.518951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.518923    2580 fs.go:135] Filesystem UUIDs: map[4e99bb30-3e47-4114-a12e-264550694727:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f074e2a4-ced7-4181-bb8b-5c205f7fa306:/dev/nvme0n1p4]
Sep 19 10:35:54.519048 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.518950    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 19 10:35:54.525397 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.525276    2580 manager.go:216] Machine: {Timestamp:2026-09-19 10:35:54.523851373 +0000 UTC m=+0.370448096 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099818 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec244ebf36259203f0ac1c01deb3c689 SystemUUID:ec244ebf-3625-9203-f0ac-1c01deb3c689 BootID:b6d047af-7b6d-4f80-8340-ff4a9c21b297 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:55:e5:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:55:e5:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:f5:9c:de:b9:31 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 19 10:35:54.525397 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.525384    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 19 10:35:54.525530 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.525469    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 19 10:35:54.528469 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.528452    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 10:35:54.529109 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.529083    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 19 10:35:54.529249 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.529114    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-220.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 19 10:35:54.530037 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.530027    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 19 10:35:54.530073 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.530040    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 19 10:35:54.530073 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.530049    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 10:35:54.530073 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.530069    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 10:35:54.530193 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.530146    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 19 10:35:54.534867 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.534855    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 10:35:54.534994 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.534985    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 19 10:35:54.535034 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.535016    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 19 10:35:54.535034 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.535025    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 19 10:35:54.535120 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.535043    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 19 10:35:54.535120 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.535056    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 19 10:35:54.535120 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.535068    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 19 10:35:54.535452 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.535441    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 10:35:54.535486 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.535460    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 10:35:54.537783 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.537767    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 19 10:35:54.540866 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.540851    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 19 10:35:54.540939 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.540882    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 19 10:35:54.543200 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.543185    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 19 10:35:54.543284 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.543207    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 19 10:35:54.543284 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.543213    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 19 10:35:54.543284 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.543219    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 19 10:35:54.543284 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.543228    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 19 10:35:54.543284 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.543233    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 19 10:35:54.543284 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.543246    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 19 10:35:54.543284 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.543251    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 19 10:35:54.543284 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.543267    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 19 10:35:54.543284 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.543273    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 19 10:35:54.543284 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.543281    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 19 10:35:54.543284 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.543288    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 19 10:35:54.544248 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.544236    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 19 10:35:54.544248 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.544247    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 19 10:35:54.545860 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.545835    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-220.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 19 10:35:54.545860 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.545833    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 19 10:35:54.548335 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.548320    2580 server.go:1268] "Started kubelet"
Sep 19 10:35:54.548484 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.548450    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 19 10:35:54.548604 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.548423    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 19 10:35:54.548654 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.548629    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 19 10:35:54.549161 ip-10-0-129-220 systemd[1]: Started Kubernetes Kubelet.
Sep 19 10:35:54.551044 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.551029    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 19 10:35:54.552681 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.552663    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 19 10:35:54.553349 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.553327    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 19 10:35:54.555296 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.555281    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-220.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 19 10:35:54.556403 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.555315    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-220.ec2.internal.18d6b27d9fd11a0e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-220.ec2.internal,UID:ip-10-0-129-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-220.ec2.internal,},FirstTimestamp:2026-09-19 10:35:54.548288014 +0000 UTC m=+0.394884737,LastTimestamp:2026-09-19 10:35:54.548288014 +0000 UTC m=+0.394884737,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-220.ec2.internal,}"
Sep 19 10:35:54.562462 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.562440    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 19 10:35:54.562462 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.562453    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 19 10:35:54.562614 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.562587    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-220.ec2.internal\" not found"
Sep 19 10:35:54.563300 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.563173    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 19 10:35:54.563300 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.563184    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 19 10:35:54.563300 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.563201    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 19 10:35:54.563300 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.563268    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 19 10:35:54.563300 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.563280    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 19 10:35:54.563816 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.563798    2580 factory.go:153] Registering CRI-O factory
Sep 19 10:35:54.563816 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.563819    2580 factory.go:223] Registration of the crio container factory successfully
Sep 19 10:35:54.563948 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.563878    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 19 10:35:54.563948 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.563888    2580 factory.go:55] Registering systemd factory
Sep 19 10:35:54.563948 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.563896    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 19 10:35:54.563948 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.563930    2580 factory.go:103] Registering Raw factory
Sep 19 10:35:54.563948 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.563949    2580 manager.go:1196] Started watching for new ooms in manager
Sep 19 10:35:54.564267 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.564240    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-220.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 19 10:35:54.564421 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.564398    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 19 10:35:54.565169 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.565157    2580 manager.go:349] Starting recovery of all containers
Sep 19 10:35:54.572273 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.572229    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 19 10:35:54.575400 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.575382    2580 manager.go:354] Recovery completed
Sep 19 10:35:54.576543 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.576522    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bsqbg"
Sep 19 10:35:54.579865 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.579851    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 10:35:54.581645 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.581628    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bsqbg"
Sep 19 10:35:54.582710 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.582695    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 10:35:54.582771 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.582723    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 10:35:54.582771 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.582734    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientPID"
Sep 19 10:35:54.583261 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.583242    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 19 10:35:54.583261 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.583255    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 19 10:35:54.583402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.583269    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 10:35:54.583402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.583302    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 10:35:54.584080 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.584064    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 10:35:54.584136 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.584091    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 10:35:54.584136 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.584103    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientPID"
Sep 19 10:35:54.585794 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.585779    2580 policy_none.go:49] "None policy: Start"
Sep 19 10:35:54.585794 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.585796    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 19 10:35:54.585884 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.585807    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 19 10:35:54.588399 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.588388    2580 policy_none.go:47] "Start"
Sep 19 10:35:54.641596 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.629519    2580 manager.go:337] "Starting Device Plugin manager"
Sep 19 10:35:54.641596 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.629559    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 19 10:35:54.641596 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.629573    2580 server.go:87] "Starting device plugin registration server"
Sep 19 10:35:54.641596 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.629825    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 19 10:35:54.641596 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.629834    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 19 10:35:54.641596 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.630000    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 19 10:35:54.641596 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.630104    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 19 10:35:54.641596 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.630113    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 19 10:35:54.641596 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.630614    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 19 10:35:54.641596 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.630671    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-220.ec2.internal\" not found"
Sep 19 10:35:54.661475 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.661442    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 19 10:35:54.661603 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.661513    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 19 10:35:54.661603 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.661544    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 19 10:35:54.661695 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.661592    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 19 10:35:54.663588 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.663569    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 10:35:54.730845 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.730763    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 10:35:54.731783 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.731765    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 10:35:54.731849 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.731803    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 10:35:54.731849 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.731814    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientPID"
Sep 19 10:35:54.731849 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.731837    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.740672 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.740656    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.740724 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.740682    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-220.ec2.internal\": node \"ip-10-0-129-220.ec2.internal\" not found"
Sep 19 10:35:54.761707 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.761670    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-220.ec2.internal"]
Sep 19 10:35:54.761843 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.761754    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 10:35:54.762635 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.762618    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 10:35:54.762733 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.762648    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 10:35:54.762733 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.762658    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientPID"
Sep 19 10:35:54.764691 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.764673    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-220.ec2.internal\" not found"
Sep 19 10:35:54.765119 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.765104    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 10:35:54.765236 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.765222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.765270 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.765260    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 10:35:54.765872 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.765851    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 10:35:54.765872 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.765866    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 10:35:54.765872 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.765875    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 10:35:54.766048 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.765885    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientPID"
Sep 19 10:35:54.766048 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.765885    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 10:35:54.766048 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.766008    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientPID"
Sep 19 10:35:54.768258 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.768244    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.768306 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.768271    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 10:35:54.769120 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.769107    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 10:35:54.769192 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.769131    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 10:35:54.769192 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.769141    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeHasSufficientPID"
Sep 19 10:35:54.799737 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.799710    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-220.ec2.internal\" not found" node="ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.804168 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.804147    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-220.ec2.internal\" not found" node="ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.864008 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.863975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5995f284b80b5abfcd665894b46e6d3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal\" (UID: \"5995f284b80b5abfcd665894b46e6d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.864147 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.864032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5995f284b80b5abfcd665894b46e6d3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal\" (UID: \"5995f284b80b5abfcd665894b46e6d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.864992 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.864974    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-220.ec2.internal\" not found"
Sep 19 10:35:54.964419 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.964388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5995f284b80b5abfcd665894b46e6d3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal\" (UID: \"5995f284b80b5abfcd665894b46e6d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.964500 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.964423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4197a5b43e365e5128cbafdb49a967bc-config\") pod \"kube-apiserver-proxy-ip-10-0-129-220.ec2.internal\" (UID: \"4197a5b43e365e5128cbafdb49a967bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.964500 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.964452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5995f284b80b5abfcd665894b46e6d3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal\" (UID: \"5995f284b80b5abfcd665894b46e6d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.964500 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.964489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5995f284b80b5abfcd665894b46e6d3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal\" (UID: \"5995f284b80b5abfcd665894b46e6d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.964500 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:54.964489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5995f284b80b5abfcd665894b46e6d3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal\" (UID: \"5995f284b80b5abfcd665894b46e6d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:54.965414 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:54.965399    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-220.ec2.internal\" not found"
Sep 19 10:35:55.065165 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.065089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4197a5b43e365e5128cbafdb49a967bc-config\") pod \"kube-apiserver-proxy-ip-10-0-129-220.ec2.internal\" (UID: \"4197a5b43e365e5128cbafdb49a967bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:55.065165 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.065162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4197a5b43e365e5128cbafdb49a967bc-config\") pod \"kube-apiserver-proxy-ip-10-0-129-220.ec2.internal\" (UID: \"4197a5b43e365e5128cbafdb49a967bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:55.066140 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:55.066118    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-220.ec2.internal\" not found"
Sep 19 10:35:55.103327 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.103291    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:55.107001 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.106982    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:55.167156 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:55.167122    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-220.ec2.internal\" not found"
Sep 19 10:35:55.267576 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:55.267545    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-220.ec2.internal\" not found"
Sep 19 10:35:55.368027 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:55.367954    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-220.ec2.internal\" not found"
Sep 19 10:35:55.468465 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:55.468433    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-220.ec2.internal\" not found"
Sep 19 10:35:55.494627 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.494593    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 19 10:35:55.494818 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.494796    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 10:35:55.568657 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:55.568486    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-220.ec2.internal\" not found"
Sep 19 10:35:55.588594 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.588554    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 10:30:55 +0000 UTC" deadline="2028-03-20 13:00:07.200675568 +0000 UTC"
Sep 19 10:35:55.588594 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.588587    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13154h24m11.61209331s"
Sep 19 10:35:55.638397 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.638315    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 10:35:55.669429 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:55.669398    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-220.ec2.internal\" not found"
Sep 19 10:35:55.690321 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.690292    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 10:35:55.756619 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.756587    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 10:35:55.757747 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:55.757713    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5995f284b80b5abfcd665894b46e6d3d.slice/crio-3df397cd31493c79d1ca3cd0ea82b826876018be51229909962708d31064e85f WatchSource:0}: Error finding container 3df397cd31493c79d1ca3cd0ea82b826876018be51229909962708d31064e85f: Status 404 returned error can't find the container with id 3df397cd31493c79d1ca3cd0ea82b826876018be51229909962708d31064e85f
Sep 19 10:35:55.758168 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:55.758145    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4197a5b43e365e5128cbafdb49a967bc.slice/crio-96979a01da87412f7ae9d831ce9f67250bee5c69abe4a2b1a1b90afb4122bd3e WatchSource:0}: Error finding container 96979a01da87412f7ae9d831ce9f67250bee5c69abe4a2b1a1b90afb4122bd3e: Status 404 returned error can't find the container with id 96979a01da87412f7ae9d831ce9f67250bee5c69abe4a2b1a1b90afb4122bd3e
Sep 19 10:35:55.762727 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.762709    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:55.764072 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.764057    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 10:35:55.776157 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.776137    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 10:35:55.776854 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.776837    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-220.ec2.internal"
Sep 19 10:35:55.784050 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:55.784032    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 10:35:57.724048 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.724013    2580 apiserver.go:52] "Watching apiserver"
Sep 19 10:35:57.732326 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.732293    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 19 10:35:57.733184 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.733151    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-bj85t","openshift-multus/network-metrics-daemon-fhc4f","openshift-network-diagnostics/network-check-target-hz8mn","kube-system/konnectivity-agent-2wmgv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal","openshift-multus/multus-additional-cni-plugins-9jvzl","openshift-network-operator/iptables-alerter-hzqfn","openshift-ovn-kubernetes/ovnkube-node-kx7wh","kube-system/global-pull-secret-syncer-wxp6f","kube-system/kube-apiserver-proxy-ip-10-0-129-220.ec2.internal","openshift-cluster-node-tuning-operator/tuned-pvr8n","openshift-dns/node-resolver-dkccw","openshift-image-registry/node-ca-49h5q"]
Sep 19 10:35:57.735922 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.735778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.740982 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.738089    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 19 10:35:57.740982 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.738311    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 19 10:35:57.740982 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.738613    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hh5jz\""
Sep 19 10:35:57.740982 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.738806    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-49h5q"
Sep 19 10:35:57.740982 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.740250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.740982 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.740855    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 19 10:35:57.740982 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.740858    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 19 10:35:57.741368 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.741122    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 19 10:35:57.741368 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.741148    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-26cw7\""
Sep 19 10:35:57.742356 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.742337    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 19 10:35:57.742519 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.742501    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2wmgv"
Sep 19 10:35:57.742594 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.742521    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 19 10:35:57.742648 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.742627    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 19 10:35:57.742797 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.742780    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 19 10:35:57.742872 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.742859    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v4cp9\""
Sep 19 10:35:57.743051 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.743031    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 19 10:35:57.744510 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.744492    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 19 10:35:57.744737 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.744710    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 19 10:35:57.745033 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.745016    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qm8ss\""
Sep 19 10:35:57.745918 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.745901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.748435 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.747981    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f6vfh\""
Sep 19 10:35:57.748435 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.748008    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 19 10:35:57.748435 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.748251    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 19 10:35:57.748435 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.748288    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 19 10:35:57.749722 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.749700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.751951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.751798    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lfszl\""
Sep 19 10:35:57.751951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.751821    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 19 10:35:57.751951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.751826    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 19 10:35:57.752152 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.752097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hzqfn"
Sep 19 10:35:57.753943 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.753917    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 19 10:35:57.754170 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.754149    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 19 10:35:57.754532 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.754514    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hbq2z\""
Sep 19 10:35:57.754615 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.754538    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 19 10:35:57.755616 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.755582    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.758226 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.758184    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 19 10:35:57.758400 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.758386    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 19 10:35:57.758799 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.758669    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ln4ff\""
Sep 19 10:35:57.758921 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.758905    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 19 10:35:57.759123 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.759093    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 19 10:35:57.759173 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.759123    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 19 10:35:57.759388 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.759373    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 19 10:35:57.759427 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.759391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:35:57.759482 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:57.759461    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:35:57.760649 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.760580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:35:57.760649 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:57.760638    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:35:57.762062 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-cnibin\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.762156 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-cni-binary-copy\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.762156 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-sysctl-d\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.762156 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-device-dir\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.762156 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-sys-fs\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.762330 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-multus-socket-dir-parent\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.762330 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-var-lib-kubelet\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.762330 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-multus-daemon-config\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.762443 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tthl\" (UniqueName: \"kubernetes.io/projected/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-kube-api-access-6tthl\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.762443 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.762443 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ed16eee-8b67-4503-9250-ccec8f679b5d-iptables-alerter-script\") pod \"iptables-alerter-hzqfn\" (UID: \"9ed16eee-8b67-4503-9250-ccec8f679b5d\") " pod="openshift-network-operator/iptables-alerter-hzqfn"
Sep 19 10:35:57.763066 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.763066 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-modprobe-d\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.763066 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.762998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-kubernetes\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.763278 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763201    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 10:35:57.763570 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-run\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.763570 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-sys\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.763570 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-var-lib-kubelet\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.763719 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-cnibin\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.763719 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-run-netns\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.763719 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-var-lib-cni-multus\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.763719 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-sysctl-conf\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.763719 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-tuned\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.763951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc5wx\" (UniqueName: \"kubernetes.io/projected/0f873690-cb8f-4e7f-8ad0-378029aae50d-kube-api-access-zc5wx\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.763951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.763951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-system-cni-dir\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.763951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpgg\" (UniqueName: \"kubernetes.io/projected/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-kube-api-access-jmpgg\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.763951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-sysconfig\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.763951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-systemd\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.763951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-etc-kubernetes\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.764297 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.763970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8b7\" (UniqueName: \"kubernetes.io/projected/b5bae54d-ce02-46ec-a38c-c43f61441d7d-kube-api-access-zn8b7\") pod \"node-ca-49h5q\" (UID: \"b5bae54d-ce02-46ec-a38c-c43f61441d7d\") " pod="openshift-image-registry/node-ca-49h5q"
Sep 19 10:35:57.764297 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-lib-modules\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.764297 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-host\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.764297 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f873690-cb8f-4e7f-8ad0-378029aae50d-tmp\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.764297 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-registration-dir\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.764507 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbgbp\" (UniqueName: \"kubernetes.io/projected/f0ed4f4b-0748-44d4-aaab-c304203945ea-kube-api-access-tbgbp\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.764507 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-multus-cni-dir\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.764507 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-cni-binary-copy\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.764841 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-var-lib-cni-bin\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.764841 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5bae54d-ce02-46ec-a38c-c43f61441d7d-host\") pod \"node-ca-49h5q\" (UID: \"b5bae54d-ce02-46ec-a38c-c43f61441d7d\") " pod="openshift-image-registry/node-ca-49h5q"
Sep 19 10:35:57.764841 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.764841 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9d113728-f674-42ed-b2d3-bb9b96e30704-agent-certs\") pod \"konnectivity-agent-2wmgv\" (UID: \"9d113728-f674-42ed-b2d3-bb9b96e30704\") " pod="kube-system/konnectivity-agent-2wmgv"
Sep 19 10:35:57.765018 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9d113728-f674-42ed-b2d3-bb9b96e30704-konnectivity-ca\") pod \"konnectivity-agent-2wmgv\" (UID: \"9d113728-f674-42ed-b2d3-bb9b96e30704\") " pod="kube-system/konnectivity-agent-2wmgv"
Sep 19 10:35:57.765018 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-socket-dir\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.765018 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.764976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-system-cni-dir\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.765098 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.765020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-hostroot\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.765098 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.765057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-run-multus-certs\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.765098 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.765085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5bae54d-ce02-46ec-a38c-c43f61441d7d-serviceca\") pod \"node-ca-49h5q\" (UID: \"b5bae54d-ce02-46ec-a38c-c43f61441d7d\") " pod="openshift-image-registry/node-ca-49h5q"
Sep 19 10:35:57.765202 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.765125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.765202 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.765174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-run-k8s-cni-cncf-io\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.765262 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.765207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-multus-conf-dir\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.765262 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.765243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ed16eee-8b67-4503-9250-ccec8f679b5d-host-slash\") pod \"iptables-alerter-hzqfn\" (UID: \"9ed16eee-8b67-4503-9250-ccec8f679b5d\") " pod="openshift-network-operator/iptables-alerter-hzqfn"
Sep 19 10:35:57.765324 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.765277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ms5\" (UniqueName: \"kubernetes.io/projected/9ed16eee-8b67-4503-9250-ccec8f679b5d-kube-api-access-s7ms5\") pod \"iptables-alerter-hzqfn\" (UID: \"9ed16eee-8b67-4503-9250-ccec8f679b5d\") " pod="openshift-network-operator/iptables-alerter-hzqfn"
Sep 19 10:35:57.766942 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.766918    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dkccw"
Sep 19 10:35:57.768957 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.768920    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 19 10:35:57.769065 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.768977    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tpjx7\""
Sep 19 10:35:57.769065 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.768921    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 19 10:35:57.769707 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.769692    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:35:57.769839 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:57.769747    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:35:57.782642 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.782618    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wdnzg"
Sep 19 10:35:57.790168 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.790142    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wdnzg"
Sep 19 10:35:57.851962 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.851930    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 19 10:35:57.854955 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.854892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-220.ec2.internal" event={"ID":"4197a5b43e365e5128cbafdb49a967bc","Type":"ContainerStarted","Data":"96979a01da87412f7ae9d831ce9f67250bee5c69abe4a2b1a1b90afb4122bd3e"}
Sep 19 10:35:57.856269 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.856242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal" event={"ID":"5995f284b80b5abfcd665894b46e6d3d","Type":"ContainerStarted","Data":"3df397cd31493c79d1ca3cd0ea82b826876018be51229909962708d31064e85f"}
Sep 19 10:35:57.866054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-run-k8s-cni-cncf-io\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.866214 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-multus-conf-dir\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.866214 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-multus-conf-dir\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.866214 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-run-k8s-cni-cncf-io\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.866214 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ms5\" (UniqueName: \"kubernetes.io/projected/9ed16eee-8b67-4503-9250-ccec8f679b5d-kube-api-access-s7ms5\") pod \"iptables-alerter-hzqfn\" (UID: \"9ed16eee-8b67-4503-9250-ccec8f679b5d\") " pod="openshift-network-operator/iptables-alerter-hzqfn"
Sep 19 10:35:57.866402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-sysctl-d\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.866402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-run-ovn\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.866402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cadc00c5-c78d-4548-a031-b59af85e69e7-ovn-node-metrics-cert\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.866402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4c1b1206-2361-42b9-b787-b49f81c893c7-hosts-file\") pod \"node-resolver-dkccw\" (UID: \"4c1b1206-2361-42b9-b787-b49f81c893c7\") " pod="openshift-dns/node-resolver-dkccw"
Sep 19 10:35:57.866402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-multus-daemon-config\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.866402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.866402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-kubernetes\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.866619 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-run\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.866619 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-sysctl-d\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.866619 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-etc-openvswitch\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.866619 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c1b1206-2361-42b9-b787-b49f81c893c7-tmp-dir\") pod \"node-resolver-dkccw\" (UID: \"4c1b1206-2361-42b9-b787-b49f81c893c7\") " pod="openshift-dns/node-resolver-dkccw"
Sep 19 10:35:57.866619 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4phcx\" (UniqueName: \"kubernetes.io/projected/4c1b1206-2361-42b9-b787-b49f81c893c7-kube-api-access-4phcx\") pod \"node-resolver-dkccw\" (UID: \"4c1b1206-2361-42b9-b787-b49f81c893c7\") " pod="openshift-dns/node-resolver-dkccw"
Sep 19 10:35:57.866619 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-kubernetes\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.866619 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ed16eee-8b67-4503-9250-ccec8f679b5d-iptables-alerter-script\") pod \"iptables-alerter-hzqfn\" (UID: \"9ed16eee-8b67-4503-9250-ccec8f679b5d\") " pod="openshift-network-operator/iptables-alerter-hzqfn"
Sep 19 10:35:57.866846 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-kubelet\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.866846 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866661    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f24a9314-df59-4a55-8d61-9b6f278c322d-kubelet-config\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:35:57.866846 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-run-netns\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.866846 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-var-lib-cni-multus\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.866846 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-tuned\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.867080 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.867080 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-cni-bin\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.867080 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.867080 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-system-cni-dir\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.867080 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-log-socket\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.867080 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.866995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-etc-kubernetes\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.867080 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8b7\" (UniqueName: \"kubernetes.io/projected/b5bae54d-ce02-46ec-a38c-c43f61441d7d-kube-api-access-zn8b7\") pod \"node-ca-49h5q\" (UID: \"b5bae54d-ce02-46ec-a38c-c43f61441d7d\") " pod="openshift-image-registry/node-ca-49h5q"
Sep 19 10:35:57.867080 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-lib-modules\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.867080 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f873690-cb8f-4e7f-8ad0-378029aae50d-tmp\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.867513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-registration-dir\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.867513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-multus-cni-dir\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.867513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867168    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.867513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-cni-binary-copy\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.867513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-run\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.867513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5bae54d-ce02-46ec-a38c-c43f61441d7d-host\") pod \"node-ca-49h5q\" (UID: \"b5bae54d-ce02-46ec-a38c-c43f61441d7d\") " pod="openshift-image-registry/node-ca-49h5q"
Sep 19 10:35:57.867513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.867513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-run-openvswitch\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.867513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-multus-cni-dir\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.867513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9d113728-f674-42ed-b2d3-bb9b96e30704-konnectivity-ca\") pod \"konnectivity-agent-2wmgv\" (UID: \"9d113728-f674-42ed-b2d3-bb9b96e30704\") " pod="kube-system/konnectivity-agent-2wmgv"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-hostroot\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-run-multus-certs\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5bae54d-ce02-46ec-a38c-c43f61441d7d-serviceca\") pod \"node-ca-49h5q\" (UID: \"b5bae54d-ce02-46ec-a38c-c43f61441d7d\") " pod="openshift-image-registry/node-ca-49h5q"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-cnibin\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-cni-binary-copy\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cadc00c5-c78d-4548-a031-b59af85e69e7-ovnkube-script-lib\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc5wx\" (UniqueName: \"kubernetes.io/projected/0f873690-cb8f-4e7f-8ad0-378029aae50d-kube-api-access-zc5wx\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ed16eee-8b67-4503-9250-ccec8f679b5d-host-slash\") pod \"iptables-alerter-hzqfn\" (UID: \"9ed16eee-8b67-4503-9250-ccec8f679b5d\") " pod="openshift-network-operator/iptables-alerter-hzqfn"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-device-dir\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-sys-fs\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-multus-socket-dir-parent\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-var-lib-kubelet\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tthl\" (UniqueName: \"kubernetes.io/projected/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-kube-api-access-6tthl\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-modprobe-d\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.867987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-slash\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.868021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-cni-netd\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9d113728-f674-42ed-b2d3-bb9b96e30704-konnectivity-ca\") pod \"konnectivity-agent-2wmgv\" (UID: \"9d113728-f674-42ed-b2d3-bb9b96e30704\") " pod="kube-system/konnectivity-agent-2wmgv"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-sys\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-var-lib-kubelet\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ed16eee-8b67-4503-9250-ccec8f679b5d-host-slash\") pod \"iptables-alerter-hzqfn\" (UID: \"9ed16eee-8b67-4503-9250-ccec8f679b5d\") " pod="openshift-network-operator/iptables-alerter-hzqfn"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-device-dir\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-cnibin\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-node-log\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bzrz\" (UniqueName: \"kubernetes.io/projected/cadc00c5-c78d-4548-a031-b59af85e69e7-kube-api-access-5bzrz\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868403    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmns2\" (UniqueName: \"kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2\") pod \"network-check-target-hz8mn\" (UID: \"291c0f79-1f63-4e4c-bcf3-6f85ee2edee6\") " pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpgg\" (UniqueName: \"kubernetes.io/projected/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-kube-api-access-jmpgg\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-sysconfig\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.868492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-systemd\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-run-systemd\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f24a9314-df59-4a55-8d61-9b6f278c322d-dbus\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrj5d\" (UniqueName: \"kubernetes.io/projected/162318c8-6ce5-4dd0-9f6c-6c61d1858816-kube-api-access-jrj5d\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-host\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cadc00c5-c78d-4548-a031-b59af85e69e7-env-overrides\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbgbp\" (UniqueName: \"kubernetes.io/projected/f0ed4f4b-0748-44d4-aaab-c304203945ea-kube-api-access-tbgbp\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-var-lib-cni-bin\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-var-lib-openvswitch\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9d113728-f674-42ed-b2d3-bb9b96e30704-agent-certs\") pod \"konnectivity-agent-2wmgv\" (UID: \"9d113728-f674-42ed-b2d3-bb9b96e30704\") " pod="kube-system/konnectivity-agent-2wmgv"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-socket-dir\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-multus-daemon-config\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-system-cni-dir\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-cni-binary-copy\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868918    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-hostroot\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.869102 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-etc-kubernetes\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-systemd-units\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-multus-socket-dir-parent\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869051    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-sys\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-sys-fs\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.868987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-run-netns\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-lib-modules\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cadc00c5-c78d-4548-a031-b59af85e69e7-ovnkube-config\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-socket-dir\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-sysctl-conf\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5bae54d-ce02-46ec-a38c-c43f61441d7d-host\") pod \"node-ca-49h5q\" (UID: \"b5bae54d-ce02-46ec-a38c-c43f61441d7d\") " pod="openshift-image-registry/node-ca-49h5q"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-var-lib-cni-bin\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869154    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0ed4f4b-0748-44d4-aaab-c304203945ea-registration-dir\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-run-multus-certs\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-sysctl-conf\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-sysconfig\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5bae54d-ce02-46ec-a38c-c43f61441d7d-serviceca\") pod \"node-ca-49h5q\" (UID: \"b5bae54d-ce02-46ec-a38c-c43f61441d7d\") " pod="openshift-image-registry/node-ca-49h5q"
Sep 19 10:35:57.869880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-host\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.870715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-cnibin\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.870715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ed16eee-8b67-4503-9250-ccec8f679b5d-iptables-alerter-script\") pod \"iptables-alerter-hzqfn\" (UID: \"9ed16eee-8b67-4503-9250-ccec8f679b5d\") " pod="openshift-network-operator/iptables-alerter-hzqfn"
Sep 19 10:35:57.870715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.870715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-cnibin\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.870715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-run-netns\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.870715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-var-lib-kubelet\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.870715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-var-lib-kubelet\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.870715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-host-var-lib-cni-multus\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.870715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-cni-binary-copy\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.870715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-systemd\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.870715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-modprobe-d\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.870715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-system-cni-dir\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.870715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.869929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-system-cni-dir\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.873775 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.873730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f873690-cb8f-4e7f-8ad0-378029aae50d-tmp\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.877983 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.877953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc5wx\" (UniqueName: \"kubernetes.io/projected/0f873690-cb8f-4e7f-8ad0-378029aae50d-kube-api-access-zc5wx\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.878586 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.878544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f873690-cb8f-4e7f-8ad0-378029aae50d-etc-tuned\") pod \"tuned-pvr8n\" (UID: \"0f873690-cb8f-4e7f-8ad0-378029aae50d\") " pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:57.880050 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.880016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tthl\" (UniqueName: \"kubernetes.io/projected/238d9df9-4e46-4f9f-bcae-6d7e4a7913d6-kube-api-access-6tthl\") pod \"multus-bj85t\" (UID: \"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6\") " pod="openshift-multus/multus-bj85t"
Sep 19 10:35:57.880319 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.880287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9d113728-f674-42ed-b2d3-bb9b96e30704-agent-certs\") pod \"konnectivity-agent-2wmgv\" (UID: \"9d113728-f674-42ed-b2d3-bb9b96e30704\") " pod="kube-system/konnectivity-agent-2wmgv"
Sep 19 10:35:57.880699 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.880666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ms5\" (UniqueName: \"kubernetes.io/projected/9ed16eee-8b67-4503-9250-ccec8f679b5d-kube-api-access-s7ms5\") pod \"iptables-alerter-hzqfn\" (UID: \"9ed16eee-8b67-4503-9250-ccec8f679b5d\") " pod="openshift-network-operator/iptables-alerter-hzqfn"
Sep 19 10:35:57.881651 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.881631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8b7\" (UniqueName: \"kubernetes.io/projected/b5bae54d-ce02-46ec-a38c-c43f61441d7d-kube-api-access-zn8b7\") pod \"node-ca-49h5q\" (UID: \"b5bae54d-ce02-46ec-a38c-c43f61441d7d\") " pod="openshift-image-registry/node-ca-49h5q"
Sep 19 10:35:57.882350 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.882282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpgg\" (UniqueName: \"kubernetes.io/projected/b8d90df9-5745-4b48-bd08-b5cdb66c5c9d-kube-api-access-jmpgg\") pod \"multus-additional-cni-plugins-9jvzl\" (UID: \"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d\") " pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:57.882955 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.882934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbgbp\" (UniqueName: \"kubernetes.io/projected/f0ed4f4b-0748-44d4-aaab-c304203945ea-kube-api-access-tbgbp\") pod \"aws-ebs-csi-driver-node-r9gb9\" (UID: \"f0ed4f4b-0748-44d4-aaab-c304203945ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:57.970270 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-run-openvswitch\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970270 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cadc00c5-c78d-4548-a031-b59af85e69e7-ovnkube-script-lib\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970498 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-slash\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970498 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-cni-netd\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970498 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970498 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:35:57.970498 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-node-log\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970498 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bzrz\" (UniqueName: \"kubernetes.io/projected/cadc00c5-c78d-4548-a031-b59af85e69e7-kube-api-access-5bzrz\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970498 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970447    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmns2\" (UniqueName: \"kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2\") pod \"network-check-target-hz8mn\" (UID: \"291c0f79-1f63-4e4c-bcf3-6f85ee2edee6\") " pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:35:57.970498 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-run-systemd\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970498 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f24a9314-df59-4a55-8d61-9b6f278c322d-dbus\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:35:57.970894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrj5d\" (UniqueName: \"kubernetes.io/projected/162318c8-6ce5-4dd0-9f6c-6c61d1858816-kube-api-access-jrj5d\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:35:57.970894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cadc00c5-c78d-4548-a031-b59af85e69e7-env-overrides\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-var-lib-openvswitch\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:35:57.970894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-systemd-units\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-run-netns\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cadc00c5-c78d-4548-a031-b59af85e69e7-ovnkube-config\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-run-ovn\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cadc00c5-c78d-4548-a031-b59af85e69e7-ovn-node-metrics-cert\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4c1b1206-2361-42b9-b787-b49f81c893c7-hosts-file\") pod \"node-resolver-dkccw\" (UID: \"4c1b1206-2361-42b9-b787-b49f81c893c7\") " pod="openshift-dns/node-resolver-dkccw"
Sep 19 10:35:57.970894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-etc-openvswitch\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.970894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c1b1206-2361-42b9-b787-b49f81c893c7-tmp-dir\") pod \"node-resolver-dkccw\" (UID: \"4c1b1206-2361-42b9-b787-b49f81c893c7\") " pod="openshift-dns/node-resolver-dkccw"
Sep 19 10:35:57.970894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f24a9314-df59-4a55-8d61-9b6f278c322d-dbus\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-run-openvswitch\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.970981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4phcx\" (UniqueName: \"kubernetes.io/projected/4c1b1206-2361-42b9-b787-b49f81c893c7-kube-api-access-4phcx\") pod \"node-resolver-dkccw\" (UID: \"4c1b1206-2361-42b9-b787-b49f81c893c7\") " pod="openshift-dns/node-resolver-dkccw"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-kubelet\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f24a9314-df59-4a55-8d61-9b6f278c322d-kubelet-config\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-cni-bin\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:57.971122    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-log-socket\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-cni-netd\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-slash\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:57.971225    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret podName:f24a9314-df59-4a55-8d61-9b6f278c322d nodeName:}" failed. No retries permitted until 2026-09-19 10:35:58.471200897 +0000 UTC m=+3.129946075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret") pod "global-pull-secret-syncer-wxp6f" (UID: "f24a9314-df59-4a55-8d61-9b6f278c322d") : object "kube-system"/"original-pull-secret" not registered
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c1b1206-2361-42b9-b787-b49f81c893c7-tmp-dir\") pod \"node-resolver-dkccw\" (UID: \"4c1b1206-2361-42b9-b787-b49f81c893c7\") " pod="openshift-dns/node-resolver-dkccw"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f24a9314-df59-4a55-8d61-9b6f278c322d-kubelet-config\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-log-socket\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-kubelet\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.971502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971296    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-cni-bin\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-var-lib-openvswitch\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-run-systemd\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-systemd-units\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-run-ovn\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-node-log\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:57.971460    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-host-run-netns\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4c1b1206-2361-42b9-b787-b49f81c893c7-hosts-file\") pod \"node-resolver-dkccw\" (UID: \"4c1b1206-2361-42b9-b787-b49f81c893c7\") " pod="openshift-dns/node-resolver-dkccw"
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:57.971516    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs podName:162318c8-6ce5-4dd0-9f6c-6c61d1858816 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:58.471501281 +0000 UTC m=+3.130246434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs") pod "network-metrics-daemon-fhc4f" (UID: "162318c8-6ce5-4dd0-9f6c-6c61d1858816") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cadc00c5-c78d-4548-a031-b59af85e69e7-ovnkube-script-lib\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cadc00c5-c78d-4548-a031-b59af85e69e7-etc-openvswitch\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cadc00c5-c78d-4548-a031-b59af85e69e7-env-overrides\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.972317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.971967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cadc00c5-c78d-4548-a031-b59af85e69e7-ovnkube-config\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.974224 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.974170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cadc00c5-c78d-4548-a031-b59af85e69e7-ovn-node-metrics-cert\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.978414 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:57.978386    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 10:35:57.978414 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:57.978414    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 10:35:57.978584 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:57.978428    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vmns2 for pod openshift-network-diagnostics/network-check-target-hz8mn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:35:57.978584 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:57.978494    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2 podName:291c0f79-1f63-4e4c-bcf3-6f85ee2edee6 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:58.478476718 +0000 UTC m=+3.137221870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vmns2" (UniqueName: "kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2") pod "network-check-target-hz8mn" (UID: "291c0f79-1f63-4e4c-bcf3-6f85ee2edee6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:35:57.980379 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.980356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bzrz\" (UniqueName: \"kubernetes.io/projected/cadc00c5-c78d-4548-a031-b59af85e69e7-kube-api-access-5bzrz\") pod \"ovnkube-node-kx7wh\" (UID: \"cadc00c5-c78d-4548-a031-b59af85e69e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:57.980656 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.980625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4phcx\" (UniqueName: \"kubernetes.io/projected/4c1b1206-2361-42b9-b787-b49f81c893c7-kube-api-access-4phcx\") pod \"node-resolver-dkccw\" (UID: \"4c1b1206-2361-42b9-b787-b49f81c893c7\") " pod="openshift-dns/node-resolver-dkccw"
Sep 19 10:35:57.984360 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:57.984340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrj5d\" (UniqueName: \"kubernetes.io/projected/162318c8-6ce5-4dd0-9f6c-6c61d1858816-kube-api-access-jrj5d\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:35:58.049331 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.049288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pvr8n"
Sep 19 10:35:58.057105 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.057081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-49h5q"
Sep 19 10:35:58.065797 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.065773    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9jvzl"
Sep 19 10:35:58.074621 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.074596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2wmgv"
Sep 19 10:35:58.080199 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.080176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9"
Sep 19 10:35:58.087921 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.087901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bj85t"
Sep 19 10:35:58.095610 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.095588    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hzqfn"
Sep 19 10:35:58.102410 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.102377    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:35:58.109010 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.108991    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dkccw"
Sep 19 10:35:58.131018 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.130988    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 10:35:58.474142 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.474041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:35:58.474142 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.474110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:35:58.474343 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:58.474217    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 10:35:58.474343 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:58.474289    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret podName:f24a9314-df59-4a55-8d61-9b6f278c322d nodeName:}" failed. No retries permitted until 2026-09-19 10:35:59.474269365 +0000 UTC m=+4.133014503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret") pod "global-pull-secret-syncer-wxp6f" (UID: "f24a9314-df59-4a55-8d61-9b6f278c322d") : object "kube-system"/"original-pull-secret" not registered
Sep 19 10:35:58.474343 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:58.474217    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:35:58.474482 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:58.474377    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs podName:162318c8-6ce5-4dd0-9f6c-6c61d1858816 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:59.474360736 +0000 UTC m=+4.133105876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs") pod "network-metrics-daemon-fhc4f" (UID: "162318c8-6ce5-4dd0-9f6c-6c61d1858816") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:35:58.574598 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.574552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmns2\" (UniqueName: \"kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2\") pod \"network-check-target-hz8mn\" (UID: \"291c0f79-1f63-4e4c-bcf3-6f85ee2edee6\") " pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:35:58.574805 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:58.574782    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 10:35:58.574905 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:58.574816    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 10:35:58.574905 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:58.574830    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vmns2 for pod openshift-network-diagnostics/network-check-target-hz8mn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:35:58.574905 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:58.574901    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2 podName:291c0f79-1f63-4e4c-bcf3-6f85ee2edee6 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:59.574880377 +0000 UTC m=+4.233625520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vmns2" (UniqueName: "kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2") pod "network-check-target-hz8mn" (UID: "291c0f79-1f63-4e4c-bcf3-6f85ee2edee6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:35:58.793375 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.793290    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 10:30:57 +0000 UTC" deadline="2028-04-20 19:37:07.507314254 +0000 UTC"
Sep 19 10:35:58.793375 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:58.793319    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13905h1m8.713999683s"
Sep 19 10:35:58.978424 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:58.978381    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d113728_f674_42ed_b2d3_bb9b96e30704.slice/crio-c28a64ce5345cebb9d30ec1c0fcf6e33ded170c48aadb0e8d58ced81dd7f90b9 WatchSource:0}: Error finding container c28a64ce5345cebb9d30ec1c0fcf6e33ded170c48aadb0e8d58ced81dd7f90b9: Status 404 returned error can't find the container with id c28a64ce5345cebb9d30ec1c0fcf6e33ded170c48aadb0e8d58ced81dd7f90b9
Sep 19 10:35:58.989509 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:58.989447    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bae54d_ce02_46ec_a38c_c43f61441d7d.slice/crio-ee59b48b766e04fb7ab38e1522984bc11c98ed728f5499fa1e51808a9f0de16b WatchSource:0}: Error finding container ee59b48b766e04fb7ab38e1522984bc11c98ed728f5499fa1e51808a9f0de16b: Status 404 returned error can't find the container with id ee59b48b766e04fb7ab38e1522984bc11c98ed728f5499fa1e51808a9f0de16b
Sep 19 10:35:58.991439 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:58.991412    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed16eee_8b67_4503_9250_ccec8f679b5d.slice/crio-14bd70a8431cb6706861e9b3cbb4fb76dff4f5e4e0e3f3cde99ee2b1eb3a96a8 WatchSource:0}: Error finding container 14bd70a8431cb6706861e9b3cbb4fb76dff4f5e4e0e3f3cde99ee2b1eb3a96a8: Status 404 returned error can't find the container with id 14bd70a8431cb6706861e9b3cbb4fb76dff4f5e4e0e3f3cde99ee2b1eb3a96a8
Sep 19 10:35:58.992951 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:58.992873    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f873690_cb8f_4e7f_8ad0_378029aae50d.slice/crio-554893b9991241498e14acb377b0ba6621fe3f24cfd2b21a909a5da64ad748d0 WatchSource:0}: Error finding container 554893b9991241498e14acb377b0ba6621fe3f24cfd2b21a909a5da64ad748d0: Status 404 returned error can't find the container with id 554893b9991241498e14acb377b0ba6621fe3f24cfd2b21a909a5da64ad748d0
Sep 19 10:35:58.993768 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:58.993729    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1b1206_2361_42b9_b787_b49f81c893c7.slice/crio-3637c04f35e4abc8f348de000c01f7a5691b7d888222b8364dc6cce178a51d4f WatchSource:0}: Error finding container 3637c04f35e4abc8f348de000c01f7a5691b7d888222b8364dc6cce178a51d4f: Status 404 returned error can't find the container with id 3637c04f35e4abc8f348de000c01f7a5691b7d888222b8364dc6cce178a51d4f
Sep 19 10:35:58.994708 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:58.994687    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ed4f4b_0748_44d4_aaab_c304203945ea.slice/crio-db254748e379bfb2050b6ab08d14b206bb06b74649a921fd053af9cb64909745 WatchSource:0}: Error finding container db254748e379bfb2050b6ab08d14b206bb06b74649a921fd053af9cb64909745: Status 404 returned error can't find the container with id db254748e379bfb2050b6ab08d14b206bb06b74649a921fd053af9cb64909745
Sep 19 10:35:58.995327 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:58.995308    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238d9df9_4e46_4f9f_bcae_6d7e4a7913d6.slice/crio-eac10f7af563ebb56f873dc7155cdf07a14ef318329049fadc0a32389948ab9b WatchSource:0}: Error finding container eac10f7af563ebb56f873dc7155cdf07a14ef318329049fadc0a32389948ab9b: Status 404 returned error can't find the container with id eac10f7af563ebb56f873dc7155cdf07a14ef318329049fadc0a32389948ab9b
Sep 19 10:35:58.996553 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:58.996497    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcadc00c5_c78d_4548_a031_b59af85e69e7.slice/crio-4f2873c0a7563461d632a526155fa741ace11c3b1c25865b868da1e2af2811b9 WatchSource:0}: Error finding container 4f2873c0a7563461d632a526155fa741ace11c3b1c25865b868da1e2af2811b9: Status 404 returned error can't find the container with id 4f2873c0a7563461d632a526155fa741ace11c3b1c25865b868da1e2af2811b9
Sep 19 10:35:58.998501 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:35:58.998442    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d90df9_5745_4b48_bd08_b5cdb66c5c9d.slice/crio-0c751fab1665dfc0feb49e796e637d81d8bd30833accbf6bc69a2c9a70cc12db WatchSource:0}: Error finding container 0c751fab1665dfc0feb49e796e637d81d8bd30833accbf6bc69a2c9a70cc12db: Status 404 returned error can't find the container with id 0c751fab1665dfc0feb49e796e637d81d8bd30833accbf6bc69a2c9a70cc12db
Sep 19 10:35:59.482717 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.482289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:35:59.482914 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.482808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:35:59.482979 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:59.482457    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 10:35:59.483148 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:59.482968    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:35:59.483148 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:59.483039    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret podName:f24a9314-df59-4a55-8d61-9b6f278c322d nodeName:}" failed. No retries permitted until 2026-09-19 10:36:01.483020511 +0000 UTC m=+6.141765664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret") pod "global-pull-secret-syncer-wxp6f" (UID: "f24a9314-df59-4a55-8d61-9b6f278c322d") : object "kube-system"/"original-pull-secret" not registered
Sep 19 10:35:59.483148 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:59.483123    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs podName:162318c8-6ce5-4dd0-9f6c-6c61d1858816 nodeName:}" failed. No retries permitted until 2026-09-19 10:36:01.483102405 +0000 UTC m=+6.141847550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs") pod "network-metrics-daemon-fhc4f" (UID: "162318c8-6ce5-4dd0-9f6c-6c61d1858816") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:35:59.584141 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.584099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmns2\" (UniqueName: \"kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2\") pod \"network-check-target-hz8mn\" (UID: \"291c0f79-1f63-4e4c-bcf3-6f85ee2edee6\") " pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:35:59.584323 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:59.584312    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 10:35:59.584388 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:59.584332    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 10:35:59.584388 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:59.584356    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vmns2 for pod openshift-network-diagnostics/network-check-target-hz8mn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:35:59.584492 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:59.584417    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2 podName:291c0f79-1f63-4e4c-bcf3-6f85ee2edee6 nodeName:}" failed. No retries permitted until 2026-09-19 10:36:01.584399491 +0000 UTC m=+6.243144631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vmns2" (UniqueName: "kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2") pod "network-check-target-hz8mn" (UID: "291c0f79-1f63-4e4c-bcf3-6f85ee2edee6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:35:59.793736 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.793559    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 10:30:57 +0000 UTC" deadline="2028-06-16 20:10:53.364234072 +0000 UTC"
Sep 19 10:35:59.793736 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.793620    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15273h34m53.570618442s"
Sep 19 10:35:59.851575 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.850853    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:35:59.851575 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:59.850980    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:35:59.851575 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.851367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:35:59.851575 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:59.851459    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:35:59.851575 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.851503    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:35:59.851575 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:35:59.851545    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:35:59.875682 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.875600    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hzqfn" event={"ID":"9ed16eee-8b67-4503-9250-ccec8f679b5d","Type":"ContainerStarted","Data":"14bd70a8431cb6706861e9b3cbb4fb76dff4f5e4e0e3f3cde99ee2b1eb3a96a8"}
Sep 19 10:35:59.892070 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.892029    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-49h5q" event={"ID":"b5bae54d-ce02-46ec-a38c-c43f61441d7d","Type":"ContainerStarted","Data":"ee59b48b766e04fb7ab38e1522984bc11c98ed728f5499fa1e51808a9f0de16b"}
Sep 19 10:35:59.907802 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.907745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9" event={"ID":"f0ed4f4b-0748-44d4-aaab-c304203945ea","Type":"ContainerStarted","Data":"db254748e379bfb2050b6ab08d14b206bb06b74649a921fd053af9cb64909745"}
Sep 19 10:35:59.914839 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.914805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-220.ec2.internal" event={"ID":"4197a5b43e365e5128cbafdb49a967bc","Type":"ContainerStarted","Data":"a58d87a0dd443369dcb06a3750162fdad49db3e2658da7fac5e8f5d2b02d73aa"}
Sep 19 10:35:59.926323 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.926286    2580 generic.go:356] "Generic (PLEG): container finished" podID="5995f284b80b5abfcd665894b46e6d3d" containerID="ffabd019275c9f592d586f179c9e15432eb75bdea58ae72ecc2446b6c45f78b2" exitCode=0
Sep 19 10:35:59.926472 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.926380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal" event={"ID":"5995f284b80b5abfcd665894b46e6d3d","Type":"ContainerDied","Data":"ffabd019275c9f592d586f179c9e15432eb75bdea58ae72ecc2446b6c45f78b2"}
Sep 19 10:35:59.928551 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.928425    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-220.ec2.internal" podStartSLOduration=3.928412625 podStartE2EDuration="3.928412625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:59.927831254 +0000 UTC m=+4.586576419" watchObservedRunningTime="2026-09-19 10:35:59.928412625 +0000 UTC m=+4.587157782"
Sep 19 10:35:59.946575 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.946529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh" event={"ID":"cadc00c5-c78d-4548-a031-b59af85e69e7","Type":"ContainerStarted","Data":"4f2873c0a7563461d632a526155fa741ace11c3b1c25865b868da1e2af2811b9"}
Sep 19 10:35:59.951318 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.951260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dkccw" event={"ID":"4c1b1206-2361-42b9-b787-b49f81c893c7","Type":"ContainerStarted","Data":"3637c04f35e4abc8f348de000c01f7a5691b7d888222b8364dc6cce178a51d4f"}
Sep 19 10:35:59.963856 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.963788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pvr8n" event={"ID":"0f873690-cb8f-4e7f-8ad0-378029aae50d","Type":"ContainerStarted","Data":"554893b9991241498e14acb377b0ba6621fe3f24cfd2b21a909a5da64ad748d0"}
Sep 19 10:35:59.971803 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.971768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2wmgv" event={"ID":"9d113728-f674-42ed-b2d3-bb9b96e30704","Type":"ContainerStarted","Data":"c28a64ce5345cebb9d30ec1c0fcf6e33ded170c48aadb0e8d58ced81dd7f90b9"}
Sep 19 10:35:59.976662 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.976627    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jvzl" event={"ID":"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d","Type":"ContainerStarted","Data":"0c751fab1665dfc0feb49e796e637d81d8bd30833accbf6bc69a2c9a70cc12db"}
Sep 19 10:35:59.987004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:35:59.986931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bj85t" event={"ID":"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6","Type":"ContainerStarted","Data":"eac10f7af563ebb56f873dc7155cdf07a14ef318329049fadc0a32389948ab9b"}
Sep 19 10:36:01.010821 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:01.010775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal" event={"ID":"5995f284b80b5abfcd665894b46e6d3d","Type":"ContainerStarted","Data":"053d91944201774ba937841c8bc91e39245fd132af8c22bacbfe9eb879734654"}
Sep 19 10:36:01.025406 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:01.025343    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-220.ec2.internal" podStartSLOduration=5.025304826 podStartE2EDuration="5.025304826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:01.025033244 +0000 UTC m=+5.683778397" watchObservedRunningTime="2026-09-19 10:36:01.025304826 +0000 UTC m=+5.684049988"
Sep 19 10:36:01.502236 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:01.502145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:01.502236 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:01.502219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:01.502514 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:01.502319    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 10:36:01.502514 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:01.502344    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:36:01.502514 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:01.502401    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs podName:162318c8-6ce5-4dd0-9f6c-6c61d1858816 nodeName:}" failed. No retries permitted until 2026-09-19 10:36:05.502383197 +0000 UTC m=+10.161128339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs") pod "network-metrics-daemon-fhc4f" (UID: "162318c8-6ce5-4dd0-9f6c-6c61d1858816") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:36:01.502514 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:01.502420    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret podName:f24a9314-df59-4a55-8d61-9b6f278c322d nodeName:}" failed. No retries permitted until 2026-09-19 10:36:05.502410597 +0000 UTC m=+10.161155742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret") pod "global-pull-secret-syncer-wxp6f" (UID: "f24a9314-df59-4a55-8d61-9b6f278c322d") : object "kube-system"/"original-pull-secret" not registered
Sep 19 10:36:01.603498 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:01.602727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmns2\" (UniqueName: \"kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2\") pod \"network-check-target-hz8mn\" (UID: \"291c0f79-1f63-4e4c-bcf3-6f85ee2edee6\") " pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:01.603498 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:01.602970    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 10:36:01.603498 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:01.602995    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 10:36:01.603498 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:01.603010    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vmns2 for pod openshift-network-diagnostics/network-check-target-hz8mn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:36:01.603498 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:01.603069    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2 podName:291c0f79-1f63-4e4c-bcf3-6f85ee2edee6 nodeName:}" failed. No retries permitted until 2026-09-19 10:36:05.603050257 +0000 UTC m=+10.261795410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vmns2" (UniqueName: "kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2") pod "network-check-target-hz8mn" (UID: "291c0f79-1f63-4e4c-bcf3-6f85ee2edee6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:36:01.851734 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:01.850894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:01.851734 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:01.851029    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:01.851734 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:01.851508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:01.851734 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:01.851595    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:01.852264 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:01.852081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:01.852264 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:01.852178    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:03.850084 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:03.850050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:03.850551 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:03.850184    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:03.850624 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:03.850557    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:03.850722 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:03.850692    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:03.850836 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:03.850820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:03.850936 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:03.850916    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:05.540688 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:05.539924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:05.540688 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:05.540009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:05.540688 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:05.540148    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 10:36:05.540688 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:05.540210    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret podName:f24a9314-df59-4a55-8d61-9b6f278c322d nodeName:}" failed. No retries permitted until 2026-09-19 10:36:13.540192191 +0000 UTC m=+18.198937346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret") pod "global-pull-secret-syncer-wxp6f" (UID: "f24a9314-df59-4a55-8d61-9b6f278c322d") : object "kube-system"/"original-pull-secret" not registered
Sep 19 10:36:05.540688 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:05.540609    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:36:05.540688 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:05.540655    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs podName:162318c8-6ce5-4dd0-9f6c-6c61d1858816 nodeName:}" failed. No retries permitted until 2026-09-19 10:36:13.54063957 +0000 UTC m=+18.199384722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs") pod "network-metrics-daemon-fhc4f" (UID: "162318c8-6ce5-4dd0-9f6c-6c61d1858816") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:36:05.640482 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:05.640442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmns2\" (UniqueName: \"kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2\") pod \"network-check-target-hz8mn\" (UID: \"291c0f79-1f63-4e4c-bcf3-6f85ee2edee6\") " pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:05.640668 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:05.640602    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 10:36:05.640668 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:05.640625    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 10:36:05.640668 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:05.640638    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vmns2 for pod openshift-network-diagnostics/network-check-target-hz8mn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:36:05.640848 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:05.640713    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2 podName:291c0f79-1f63-4e4c-bcf3-6f85ee2edee6 nodeName:}" failed. No retries permitted until 2026-09-19 10:36:13.640696457 +0000 UTC m=+18.299441597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vmns2" (UniqueName: "kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2") pod "network-check-target-hz8mn" (UID: "291c0f79-1f63-4e4c-bcf3-6f85ee2edee6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:36:05.850863 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:05.850779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:05.851302 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:05.850850    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:05.851302 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:05.851118    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:05.851302 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:05.850932    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:05.851302 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:05.851248    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:05.851585 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:05.851362    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:07.850249 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:07.850211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:07.850675 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:07.850211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:07.850675 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:07.850342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:07.850675 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:07.850338    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:07.850675 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:07.850438    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:07.850675 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:07.850544    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:09.850098 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:09.850057    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:09.850535 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:09.850108    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:09.850535 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:09.850070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:09.850535 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:09.850234    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:09.850535 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:09.850295    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:09.850535 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:09.850384    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:11.850495 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:11.850458    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:11.850986 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:11.850459    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:11.850986 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:11.850651    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:11.851100 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:11.851019    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:11.851154 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:11.851118    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:11.851213 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:11.851183    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:13.600428 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:13.600380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:13.600906 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:13.600510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:13.600906 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:13.600556    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:36:13.600906 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:13.600621    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 10:36:13.600906 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:13.600635    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs podName:162318c8-6ce5-4dd0-9f6c-6c61d1858816 nodeName:}" failed. No retries permitted until 2026-09-19 10:36:29.600614526 +0000 UTC m=+34.259359678 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs") pod "network-metrics-daemon-fhc4f" (UID: "162318c8-6ce5-4dd0-9f6c-6c61d1858816") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:36:13.600906 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:13.600682    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret podName:f24a9314-df59-4a55-8d61-9b6f278c322d nodeName:}" failed. No retries permitted until 2026-09-19 10:36:29.600665664 +0000 UTC m=+34.259410812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret") pod "global-pull-secret-syncer-wxp6f" (UID: "f24a9314-df59-4a55-8d61-9b6f278c322d") : object "kube-system"/"original-pull-secret" not registered
Sep 19 10:36:13.701838 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:13.701794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmns2\" (UniqueName: \"kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2\") pod \"network-check-target-hz8mn\" (UID: \"291c0f79-1f63-4e4c-bcf3-6f85ee2edee6\") " pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:13.702022 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:13.701914    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 10:36:13.702022 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:13.701941    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 10:36:13.702022 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:13.701954    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vmns2 for pod openshift-network-diagnostics/network-check-target-hz8mn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:36:13.702022 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:13.702008    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2 podName:291c0f79-1f63-4e4c-bcf3-6f85ee2edee6 nodeName:}" failed. No retries permitted until 2026-09-19 10:36:29.701995249 +0000 UTC m=+34.360740387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vmns2" (UniqueName: "kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2") pod "network-check-target-hz8mn" (UID: "291c0f79-1f63-4e4c-bcf3-6f85ee2edee6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:36:13.850060 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:13.850020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:13.850236 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:13.850075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:13.850236 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:13.850020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:13.850236 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:13.850161    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:13.850467 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:13.850269    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:13.850467 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:13.850378    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:15.853693 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:15.853663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:15.854246 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:15.853680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:15.854410 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:15.854261    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:15.854410 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:15.853744    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:15.854410 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:15.854362    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:15.854551 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:15.854419    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:17.046612 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.046341    2580 generic.go:356] "Generic (PLEG): container finished" podID="b8d90df9-5745-4b48-bd08-b5cdb66c5c9d" containerID="7c94e36f9f26b5c162ec5c13b866fbd14c967e4b31c2e6dbada232f490757a23" exitCode=0
Sep 19 10:36:17.046612 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.046432    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jvzl" event={"ID":"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d","Type":"ContainerDied","Data":"7c94e36f9f26b5c162ec5c13b866fbd14c967e4b31c2e6dbada232f490757a23"}
Sep 19 10:36:17.048465 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.048437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bj85t" event={"ID":"238d9df9-4e46-4f9f-bcae-6d7e4a7913d6","Type":"ContainerStarted","Data":"682e1d63ec8ef09bb5aa51cd4aa6f0cc2cd978bd59944d26ba2abe90956944cb"}
Sep 19 10:36:17.050131 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.050104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-49h5q" event={"ID":"b5bae54d-ce02-46ec-a38c-c43f61441d7d","Type":"ContainerStarted","Data":"a5e4a50aa24162cc2c57b7d25a66a243185192daee6f82fbed00cfa44582de67"}
Sep 19 10:36:17.052308 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.052288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9" event={"ID":"f0ed4f4b-0748-44d4-aaab-c304203945ea","Type":"ContainerStarted","Data":"b379c671a3dcdec283508729089f6a05282638bd80379074fea026b9df63404f"}
Sep 19 10:36:17.056319 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.055316    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 10:36:17.056319 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.055851    2580 generic.go:356] "Generic (PLEG): container finished" podID="cadc00c5-c78d-4548-a031-b59af85e69e7" containerID="eb7e7e4c1222123246b12cc9bbe3156d51600d8636ad936e85bcb2621a40b45b" exitCode=1
Sep 19 10:36:17.056319 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.055922    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh" event={"ID":"cadc00c5-c78d-4548-a031-b59af85e69e7","Type":"ContainerStarted","Data":"441e176c98958bdd249d271eab942337f3be4d9f6c26220582a85e6ae84b8671"}
Sep 19 10:36:17.056319 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.055989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh" event={"ID":"cadc00c5-c78d-4548-a031-b59af85e69e7","Type":"ContainerDied","Data":"eb7e7e4c1222123246b12cc9bbe3156d51600d8636ad936e85bcb2621a40b45b"}
Sep 19 10:36:17.056319 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.056036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh" event={"ID":"cadc00c5-c78d-4548-a031-b59af85e69e7","Type":"ContainerStarted","Data":"0205f2ab265b3e33705918840019406cc6d75f3f685a7af5a69ef96b02ef805e"}
Sep 19 10:36:17.058585 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.058559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dkccw" event={"ID":"4c1b1206-2361-42b9-b787-b49f81c893c7","Type":"ContainerStarted","Data":"31be96f1a1f5f27b84f0786a7f18847a86f26fbf311478d15e195e48fd69473f"}
Sep 19 10:36:17.060040 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.060019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pvr8n" event={"ID":"0f873690-cb8f-4e7f-8ad0-378029aae50d","Type":"ContainerStarted","Data":"026bb12ef2e928471ce89fc6a83a1fa18e28dca74d9b8a4e532a3785c08d5e8a"}
Sep 19 10:36:17.061435 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.061416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2wmgv" event={"ID":"9d113728-f674-42ed-b2d3-bb9b96e30704","Type":"ContainerStarted","Data":"45ffb541dd617de5c6b0248e43e40f8b7c30ce7a9b9de5eceaf000ce1a53a32d"}
Sep 19 10:36:17.090325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.090277    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pvr8n" podStartSLOduration=4.672956044 podStartE2EDuration="22.090260626s" totalImagesPullingTime="17.417304582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:17.07819733 +0000 UTC m=+21.736942490" watchObservedRunningTime="2026-09-19 10:36:17.090260626 +0000 UTC m=+21.749005783"
Sep 19 10:36:17.090429 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.090411    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-49h5q" podStartSLOduration=4.669980367 podStartE2EDuration="22.090403918s" totalImagesPullingTime="17.420423551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:17.089923942 +0000 UTC m=+21.748669102" watchObservedRunningTime="2026-09-19 10:36:17.090403918 +0000 UTC m=+21.749149078"
Sep 19 10:36:17.104569 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.104524    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bj85t" podStartSLOduration=3.36152348 podStartE2EDuration="21.104508305s" totalImagesPullingTime="17.742984825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:17.104201714 +0000 UTC m=+21.762946874" watchObservedRunningTime="2026-09-19 10:36:17.104508305 +0000 UTC m=+21.763253464"
Sep 19 10:36:17.123793 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.123485    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dkccw" podStartSLOduration=3.708599233 podStartE2EDuration="21.123462032s" totalImagesPullingTime="17.414862799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:17.122997988 +0000 UTC m=+21.781743148" watchObservedRunningTime="2026-09-19 10:36:17.123462032 +0000 UTC m=+21.782207195"
Sep 19 10:36:17.141729 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.141615    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2wmgv" podStartSLOduration=7.507207406 podStartE2EDuration="21.141593551s" totalImagesPullingTime="13.634386145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:17.141430999 +0000 UTC m=+21.800176158" watchObservedRunningTime="2026-09-19 10:36:17.141593551 +0000 UTC m=+21.800338712"
Sep 19 10:36:17.850253 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.850170    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:17.850253 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.850186    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:17.850471 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:17.850299    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:17.850471 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:17.850351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:17.850471 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:17.850447    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:17.850613 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:17.850541    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:18.064674 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:18.064650    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 19 10:36:18.066649 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:18.066620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hzqfn" event={"ID":"9ed16eee-8b67-4503-9250-ccec8f679b5d","Type":"ContainerStarted","Data":"ee3446bf135d01f1cc0c647c074a7f288cdefff8f64c00f68fcd978cc49e48bc"}
Sep 19 10:36:18.068821 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:18.068797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9" event={"ID":"f0ed4f4b-0748-44d4-aaab-c304203945ea","Type":"ContainerStarted","Data":"5a303ea4e6cce462d5cb7fcf5dcf0490d5b7f6e578470b19690cd42da8f45a11"}
Sep 19 10:36:18.072046 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:18.072025    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 10:36:18.072525 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:18.072490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh" event={"ID":"cadc00c5-c78d-4548-a031-b59af85e69e7","Type":"ContainerStarted","Data":"25e13375e8df0cadb63c9dfd7cbec4d5df7ff05fa6c27d437aa2ce89cc042fbb"}
Sep 19 10:36:18.072601 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:18.072576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh" event={"ID":"cadc00c5-c78d-4548-a031-b59af85e69e7","Type":"ContainerStarted","Data":"53bdcd23a32ded9e04162466ab42cb24b5b18d805aeb446ca8208c27a6abc70a"}
Sep 19 10:36:18.072641 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:18.072619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh" event={"ID":"cadc00c5-c78d-4548-a031-b59af85e69e7","Type":"ContainerStarted","Data":"affe6f59738a6b661444e32bca36a7686cb3d747bcc0ef5373cea78e0438a4a6"}
Sep 19 10:36:18.081722 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:18.081664    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hzqfn" podStartSLOduration=4.65796099 podStartE2EDuration="22.081643517s" totalImagesPullingTime="17.423682527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:18.081056784 +0000 UTC m=+22.739801943" watchObservedRunningTime="2026-09-19 10:36:18.081643517 +0000 UTC m=+22.740388677"
Sep 19 10:36:18.831112 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:18.831001    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T10:36:18.064667328Z","UUID":"912d9bae-a8bb-4f57-8834-b4ccdbf86d91","Handler":null,"Name":"","Endpoint":""}
Sep 19 10:36:18.832724 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:18.832704    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 19 10:36:18.832724 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:18.832738    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 19 10:36:19.850089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:19.850051    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:19.850089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:19.850093    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:19.850578 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:19.850049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:19.850578 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:19.850197    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:19.850578 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:19.850258    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:19.850578 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:19.850354    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:20.083406 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:20.083378    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 10:36:20.083904 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:20.083871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh" event={"ID":"cadc00c5-c78d-4548-a031-b59af85e69e7","Type":"ContainerStarted","Data":"ab28ebd575eff00601ca3746e9723af69158cd663568eb4b3519eb445b3d2f60"}
Sep 19 10:36:20.086530 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:20.086503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9" event={"ID":"f0ed4f4b-0748-44d4-aaab-c304203945ea","Type":"ContainerStarted","Data":"60ef58c4feb5a05ce41623b17351b826f6545c28a718120053a651ade711e9db"}
Sep 19 10:36:20.101943 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:20.101785    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r9gb9" podStartSLOduration=4.355052265 podStartE2EDuration="24.101764583s" totalImagesPullingTime="19.746712318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:35:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:20.100713395 +0000 UTC m=+24.759458555" watchObservedRunningTime="2026-09-19 10:36:20.101764583 +0000 UTC m=+24.760509734"
Sep 19 10:36:21.659519 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:21.659477    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2wmgv"
Sep 19 10:36:21.660530 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:21.660509    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2wmgv"
Sep 19 10:36:21.852419 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:21.852253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:21.852572 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:21.852492    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:21.852933 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:21.852914    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:21.853025 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:21.853006    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:21.853082 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:21.853069    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:21.853163 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:21.853146    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:22.092622 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:22.092588    2580 generic.go:356] "Generic (PLEG): container finished" podID="b8d90df9-5745-4b48-bd08-b5cdb66c5c9d" containerID="814d3220bc33d1234a90db26994d09e9808b34947399c1045f0b3e1e3e7b3287" exitCode=0
Sep 19 10:36:22.092783 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:22.092674    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jvzl" event={"ID":"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d","Type":"ContainerDied","Data":"814d3220bc33d1234a90db26994d09e9808b34947399c1045f0b3e1e3e7b3287"}
Sep 19 10:36:22.096801 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:22.096785    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 10:36:22.097190 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:22.097172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh" event={"ID":"cadc00c5-c78d-4548-a031-b59af85e69e7","Type":"ContainerStarted","Data":"7f0ec3a5c68ca65a0fd9da768c1c9ae9580ac2822c7c67929a847f04a809d802"}
Sep 19 10:36:22.097525 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:22.097500    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2wmgv"
Sep 19 10:36:22.097619 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:22.097556    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:36:22.097619 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:22.097595    2580 scope.go:117] "RemoveContainer" containerID="eb7e7e4c1222123246b12cc9bbe3156d51600d8636ad936e85bcb2621a40b45b"
Sep 19 10:36:22.097619 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:22.097606    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:36:22.097769 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:22.097654    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:36:22.097976 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:22.097958    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2wmgv"
Sep 19 10:36:22.114991 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:22.114967    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:36:22.115093 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:22.115078    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:36:23.113837 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:23.102065    2580 generic.go:356] "Generic (PLEG): container finished" podID="b8d90df9-5745-4b48-bd08-b5cdb66c5c9d" containerID="1394a4ff953fff8e8cb02ea98e5ebf64f36b20f6c207f47149187220d1d39bb8" exitCode=0
Sep 19 10:36:23.113837 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:23.102735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jvzl" event={"ID":"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d","Type":"ContainerDied","Data":"1394a4ff953fff8e8cb02ea98e5ebf64f36b20f6c207f47149187220d1d39bb8"}
Sep 19 10:36:23.113837 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:23.111870    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 10:36:23.113837 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:23.112485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh" event={"ID":"cadc00c5-c78d-4548-a031-b59af85e69e7","Type":"ContainerStarted","Data":"7d41d6424ab10998d21aece8c1c0fe5f490ef0eaa5a99b1b482f81ef9a1d60e0"}
Sep 19 10:36:23.149541 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:23.149473    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh" podStartSLOduration=9.723613365 podStartE2EDuration="27.149449682s" totalImagesPullingTime="17.425836317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:23.1478712 +0000 UTC m=+27.806616350" watchObservedRunningTime="2026-09-19 10:36:23.149449682 +0000 UTC m=+27.808194843"
Sep 19 10:36:23.773004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:23.772200    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hz8mn"]
Sep 19 10:36:23.773004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:23.772543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:23.773004 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:23.772650    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:23.775146 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:23.775115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wxp6f"]
Sep 19 10:36:23.775251 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:23.775235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:23.775344 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:23.775316    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:23.788470 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:23.788413    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fhc4f"]
Sep 19 10:36:23.788606 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:23.788583    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:23.788720 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:23.788693    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:24.117658 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:24.117558    2580 generic.go:356] "Generic (PLEG): container finished" podID="b8d90df9-5745-4b48-bd08-b5cdb66c5c9d" containerID="62fe57c19762b9083a525a4d5ff36e4ce73e0082f96b174aa9cdfbec15726567" exitCode=0
Sep 19 10:36:24.118114 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:24.117630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jvzl" event={"ID":"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d","Type":"ContainerDied","Data":"62fe57c19762b9083a525a4d5ff36e4ce73e0082f96b174aa9cdfbec15726567"}
Sep 19 10:36:25.850975 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:25.850813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:25.850975 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:25.850932    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:25.851536 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:25.850977    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:25.851536 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:25.851064    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:25.851536 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:25.851232    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:25.851536 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:25.851313    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:27.849960 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:27.849865    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:27.850402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:27.849865    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:27.850402 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:27.849993    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hz8mn" podUID="291c0f79-1f63-4e4c-bcf3-6f85ee2edee6"
Sep 19 10:36:27.850402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:27.850008    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:27.850402 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:27.850114    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxp6f" podUID="f24a9314-df59-4a55-8d61-9b6f278c322d"
Sep 19 10:36:27.850402 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:27.850199    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fhc4f" podUID="162318c8-6ce5-4dd0-9f6c-6c61d1858816"
Sep 19 10:36:29.132825 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.132546    2580 generic.go:356] "Generic (PLEG): container finished" podID="b8d90df9-5745-4b48-bd08-b5cdb66c5c9d" containerID="062ac871174e8ad8db41f3a91653a6b089a4b2574d2bd4697b69e4046fa506a5" exitCode=0
Sep 19 10:36:29.132825 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.132620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jvzl" event={"ID":"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d","Type":"ContainerDied","Data":"062ac871174e8ad8db41f3a91653a6b089a4b2574d2bd4697b69e4046fa506a5"}
Sep 19 10:36:29.613723 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.613683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:29.613899 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.613738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:29.613899 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:29.613857    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:36:29.613899 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:29.613858    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 10:36:29.614000 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:29.613913    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs podName:162318c8-6ce5-4dd0-9f6c-6c61d1858816 nodeName:}" failed. No retries permitted until 2026-09-19 10:37:01.613898829 +0000 UTC m=+66.272643979 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs") pod "network-metrics-daemon-fhc4f" (UID: "162318c8-6ce5-4dd0-9f6c-6c61d1858816") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 10:36:29.614000 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:29.613926    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret podName:f24a9314-df59-4a55-8d61-9b6f278c322d nodeName:}" failed. No retries permitted until 2026-09-19 10:37:01.613919962 +0000 UTC m=+66.272665100 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret") pod "global-pull-secret-syncer-wxp6f" (UID: "f24a9314-df59-4a55-8d61-9b6f278c322d") : object "kube-system"/"original-pull-secret" not registered
Sep 19 10:36:29.627319 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.627282    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-220.ec2.internal" event="NodeReady"
Sep 19 10:36:29.627440 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.627428    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 19 10:36:29.658516 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.658476    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"]
Sep 19 10:36:29.674224 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.674187    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69f679b797-cmc99"]
Sep 19 10:36:29.674396 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.674337    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.676588 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.676564    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 19 10:36:29.676588 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.676578    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 19 10:36:29.676826 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.676559    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 19 10:36:29.676952 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.676932    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 19 10:36:29.677405 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.677357    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 19 10:36:29.677860 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.677827    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 19 10:36:29.678419 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.678364    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 19 10:36:29.690375 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.690349    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"]
Sep 19 10:36:29.690513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.690478    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.692574 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.692545    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 19 10:36:29.692858 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.692838    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 19 10:36:29.692957 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.692838    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lp9gc\""
Sep 19 10:36:29.692957 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.692879    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 19 10:36:29.698970 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.698948    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 19 10:36:29.706676 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.706652    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn"]
Sep 19 10:36:29.706814 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.706794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"
Sep 19 10:36:29.708695 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.708677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 19 10:36:29.714385 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.714359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmns2\" (UniqueName: \"kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2\") pod \"network-check-target-hz8mn\" (UID: \"291c0f79-1f63-4e4c-bcf3-6f85ee2edee6\") " pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:29.714505 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:29.714488    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 10:36:29.714560 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:29.714511    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 10:36:29.714560 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:29.714523    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vmns2 for pod openshift-network-diagnostics/network-check-target-hz8mn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:36:29.714656 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:29.714586    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2 podName:291c0f79-1f63-4e4c-bcf3-6f85ee2edee6 nodeName:}" failed. No retries permitted until 2026-09-19 10:37:01.714569832 +0000 UTC m=+66.373314986 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vmns2" (UniqueName: "kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2") pod "network-check-target-hz8mn" (UID: "291c0f79-1f63-4e4c-bcf3-6f85ee2edee6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 10:36:29.727229 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.727203    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"]
Sep 19 10:36:29.727388 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.727235    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ncb5v"]
Sep 19 10:36:29.727388 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.727322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn"
Sep 19 10:36:29.729405 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.729386    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 19 10:36:29.729465 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.729443    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-28r95\""
Sep 19 10:36:29.745668 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.745640    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn"]
Sep 19 10:36:29.745668 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.745671    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"]
Sep 19 10:36:29.745922 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.745686    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69f679b797-cmc99"]
Sep 19 10:36:29.745922 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.745702    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mjqjg"]
Sep 19 10:36:29.745922 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.745780    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:29.747777 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.747738    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 19 10:36:29.747923 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.747907    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 19 10:36:29.748038 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.748023    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qlvz2\""
Sep 19 10:36:29.763943 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.763891    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ncb5v"]
Sep 19 10:36:29.764373 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.764349    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mjqjg"]
Sep 19 10:36:29.764475 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.764005    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mjqjg"
Sep 19 10:36:29.767006 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.766982    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 19 10:36:29.767459 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.767380    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 19 10:36:29.767459 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.767439    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-62jc5\""
Sep 19 10:36:29.767459 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.767441    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 19 10:36:29.815793 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.815693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vszqz\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-kube-api-access-vszqz\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.815793 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.815733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12123b50-d178-4a45-aab4-1785cba2616f-config-volume\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:29.815793 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.815791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/46872ad2-8593-4f10-8d29-defbae3b3c50-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.816010 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.815816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1181bcde-49d9-4582-9e19-c9603fc3c7cd-installation-pull-secrets\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.816010 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.815855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1181bcde-49d9-4582-9e19-c9603fc3c7cd-ca-trust-extracted\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.816010 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.815871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rb4z\" (UniqueName: \"kubernetes.io/projected/3ae7814e-ecb2-4a33-91d9-023cc8269455-kube-api-access-8rb4z\") pod \"managed-serviceaccount-addon-agent-76967986bd-krjfn\" (UID: \"3ae7814e-ecb2-4a33-91d9-023cc8269455\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn"
Sep 19 10:36:29.816010 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.815914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:29.816010 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.815962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9rgk\" (UniqueName: \"kubernetes.io/projected/97db0dbe-bb49-4a08-99dc-0766b211bfbe-kube-api-access-b9rgk\") pod \"ingress-canary-mjqjg\" (UID: \"97db0dbe-bb49-4a08-99dc-0766b211bfbe\") " pod="openshift-ingress-canary/ingress-canary-mjqjg"
Sep 19 10:36:29.816010 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.815993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1181bcde-49d9-4582-9e19-c9603fc3c7cd-image-registry-private-configuration\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.816010 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1181bcde-49d9-4582-9e19-c9603fc3c7cd-trusted-ca\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.816216 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-bound-sa-token\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.816216 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/494a2207-e136-4cc8-8e00-432f60322e67-tmp\") pod \"klusterlet-addon-workmgr-6b4775464-57vzm\" (UID: \"494a2207-e136-4cc8-8e00-432f60322e67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"
Sep 19 10:36:29.816216 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/46872ad2-8593-4f10-8d29-defbae3b3c50-ca\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.816216 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46872ad2-8593-4f10-8d29-defbae3b3c50-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.816216 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97db0dbe-bb49-4a08-99dc-0766b211bfbe-cert\") pod \"ingress-canary-mjqjg\" (UID: \"97db0dbe-bb49-4a08-99dc-0766b211bfbe\") " pod="openshift-ingress-canary/ingress-canary-mjqjg"
Sep 19 10:36:29.816216 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.816216 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-certificates\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.816216 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/494a2207-e136-4cc8-8e00-432f60322e67-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b4775464-57vzm\" (UID: \"494a2207-e136-4cc8-8e00-432f60322e67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"
Sep 19 10:36:29.816480 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12123b50-d178-4a45-aab4-1785cba2616f-tmp-dir\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:29.816480 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcxm6\" (UniqueName: \"kubernetes.io/projected/12123b50-d178-4a45-aab4-1785cba2616f-kube-api-access-dcxm6\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:29.816480 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ae7814e-ecb2-4a33-91d9-023cc8269455-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76967986bd-krjfn\" (UID: \"3ae7814e-ecb2-4a33-91d9-023cc8269455\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn"
Sep 19 10:36:29.816480 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/46872ad2-8593-4f10-8d29-defbae3b3c50-hub\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.816480 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz657\" (UniqueName: \"kubernetes.io/projected/494a2207-e136-4cc8-8e00-432f60322e67-kube-api-access-qz657\") pod \"klusterlet-addon-workmgr-6b4775464-57vzm\" (UID: \"494a2207-e136-4cc8-8e00-432f60322e67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"
Sep 19 10:36:29.816480 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/46872ad2-8593-4f10-8d29-defbae3b3c50-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.816480 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.816413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwgnw\" (UniqueName: \"kubernetes.io/projected/46872ad2-8593-4f10-8d29-defbae3b3c50-kube-api-access-pwgnw\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.850026 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.849987    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:36:29.850208 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.850033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:36:29.850208 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.849982    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:36:29.852200 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.852154    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 10:36:29.852387 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.852233    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ql7r7\""
Sep 19 10:36:29.852387 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.852278    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 10:36:29.852602 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.852586    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 10:36:29.852656 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.852607    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 10:36:29.852656 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.852587    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6z24q\""
Sep 19 10:36:29.916746 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.916711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.916746 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.916743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-certificates\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.916782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/494a2207-e136-4cc8-8e00-432f60322e67-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b4775464-57vzm\" (UID: \"494a2207-e136-4cc8-8e00-432f60322e67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.916805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12123b50-d178-4a45-aab4-1785cba2616f-tmp-dir\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.916821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcxm6\" (UniqueName: \"kubernetes.io/projected/12123b50-d178-4a45-aab4-1785cba2616f-kube-api-access-dcxm6\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.916845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ae7814e-ecb2-4a33-91d9-023cc8269455-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76967986bd-krjfn\" (UID: \"3ae7814e-ecb2-4a33-91d9-023cc8269455\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn"
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.916870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/46872ad2-8593-4f10-8d29-defbae3b3c50-hub\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:29.916882    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.916898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz657\" (UniqueName: \"kubernetes.io/projected/494a2207-e136-4cc8-8e00-432f60322e67-kube-api-access-qz657\") pod \"klusterlet-addon-workmgr-6b4775464-57vzm\" (UID: \"494a2207-e136-4cc8-8e00-432f60322e67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:29.916904    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f679b797-cmc99: secret "image-registry-tls" not found
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.916925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/46872ad2-8593-4f10-8d29-defbae3b3c50-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.916947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwgnw\" (UniqueName: \"kubernetes.io/projected/46872ad2-8593-4f10-8d29-defbae3b3c50-kube-api-access-pwgnw\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:29.916989    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls podName:1181bcde-49d9-4582-9e19-c9603fc3c7cd nodeName:}" failed. No retries permitted until 2026-09-19 10:36:30.416969408 +0000 UTC m=+35.075714547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls") pod "image-registry-69f679b797-cmc99" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd") : secret "image-registry-tls" not found
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vszqz\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-kube-api-access-vszqz\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12123b50-d178-4a45-aab4-1785cba2616f-config-volume\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:29.917054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/46872ad2-8593-4f10-8d29-defbae3b3c50-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.917709 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12123b50-d178-4a45-aab4-1785cba2616f-tmp-dir\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:29.917709 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-certificates\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.917709 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1181bcde-49d9-4582-9e19-c9603fc3c7cd-installation-pull-secrets\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.917709 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1181bcde-49d9-4582-9e19-c9603fc3c7cd-ca-trust-extracted\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.917709 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rb4z\" (UniqueName: \"kubernetes.io/projected/3ae7814e-ecb2-4a33-91d9-023cc8269455-kube-api-access-8rb4z\") pod \"managed-serviceaccount-addon-agent-76967986bd-krjfn\" (UID: \"3ae7814e-ecb2-4a33-91d9-023cc8269455\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn"
Sep 19 10:36:29.917709 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:29.917709 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9rgk\" (UniqueName: \"kubernetes.io/projected/97db0dbe-bb49-4a08-99dc-0766b211bfbe-kube-api-access-b9rgk\") pod \"ingress-canary-mjqjg\" (UID: \"97db0dbe-bb49-4a08-99dc-0766b211bfbe\") " pod="openshift-ingress-canary/ingress-canary-mjqjg"
Sep 19 10:36:29.917709 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1181bcde-49d9-4582-9e19-c9603fc3c7cd-image-registry-private-configuration\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.917709 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1181bcde-49d9-4582-9e19-c9603fc3c7cd-trusted-ca\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.917709 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-bound-sa-token\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.918163 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/494a2207-e136-4cc8-8e00-432f60322e67-tmp\") pod \"klusterlet-addon-workmgr-6b4775464-57vzm\" (UID: \"494a2207-e136-4cc8-8e00-432f60322e67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"
Sep 19 10:36:29.918163 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/46872ad2-8593-4f10-8d29-defbae3b3c50-ca\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.918163 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.917966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12123b50-d178-4a45-aab4-1785cba2616f-config-volume\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:29.918163 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:29.918056    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 10:36:29.918163 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:29.918118    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls podName:12123b50-d178-4a45-aab4-1785cba2616f nodeName:}" failed. No retries permitted until 2026-09-19 10:36:30.418104408 +0000 UTC m=+35.076849560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls") pod "dns-default-ncb5v" (UID: "12123b50-d178-4a45-aab4-1785cba2616f") : secret "dns-default-metrics-tls" not found
Sep 19 10:36:29.918871 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.918372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1181bcde-49d9-4582-9e19-c9603fc3c7cd-ca-trust-extracted\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.918871 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.918395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46872ad2-8593-4f10-8d29-defbae3b3c50-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.918871 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.918430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97db0dbe-bb49-4a08-99dc-0766b211bfbe-cert\") pod \"ingress-canary-mjqjg\" (UID: \"97db0dbe-bb49-4a08-99dc-0766b211bfbe\") " pod="openshift-ingress-canary/ingress-canary-mjqjg"
Sep 19 10:36:29.919136 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.919110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1181bcde-49d9-4582-9e19-c9603fc3c7cd-trusted-ca\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.920203 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.920171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/494a2207-e136-4cc8-8e00-432f60322e67-tmp\") pod \"klusterlet-addon-workmgr-6b4775464-57vzm\" (UID: \"494a2207-e136-4cc8-8e00-432f60322e67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"
Sep 19 10:36:29.920309 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.920273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/46872ad2-8593-4f10-8d29-defbae3b3c50-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.922642 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.922464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/46872ad2-8593-4f10-8d29-defbae3b3c50-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.922642 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.922476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97db0dbe-bb49-4a08-99dc-0766b211bfbe-cert\") pod \"ingress-canary-mjqjg\" (UID: \"97db0dbe-bb49-4a08-99dc-0766b211bfbe\") " pod="openshift-ingress-canary/ingress-canary-mjqjg"
Sep 19 10:36:29.922642 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.922584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ae7814e-ecb2-4a33-91d9-023cc8269455-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76967986bd-krjfn\" (UID: \"3ae7814e-ecb2-4a33-91d9-023cc8269455\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn"
Sep 19 10:36:29.922896 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.922777    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1181bcde-49d9-4582-9e19-c9603fc3c7cd-installation-pull-secrets\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.922896 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.922867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/46872ad2-8593-4f10-8d29-defbae3b3c50-ca\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.923015 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.922993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/494a2207-e136-4cc8-8e00-432f60322e67-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b4775464-57vzm\" (UID: \"494a2207-e136-4cc8-8e00-432f60322e67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"
Sep 19 10:36:29.923068 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.923033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/46872ad2-8593-4f10-8d29-defbae3b3c50-hub\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.923689 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.923649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1181bcde-49d9-4582-9e19-c9603fc3c7cd-image-registry-private-configuration\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.924496 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.924469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46872ad2-8593-4f10-8d29-defbae3b3c50-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.926005 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.925985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz657\" (UniqueName: \"kubernetes.io/projected/494a2207-e136-4cc8-8e00-432f60322e67-kube-api-access-qz657\") pod \"klusterlet-addon-workmgr-6b4775464-57vzm\" (UID: \"494a2207-e136-4cc8-8e00-432f60322e67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"
Sep 19 10:36:29.926463 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.926443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rb4z\" (UniqueName: \"kubernetes.io/projected/3ae7814e-ecb2-4a33-91d9-023cc8269455-kube-api-access-8rb4z\") pod \"managed-serviceaccount-addon-agent-76967986bd-krjfn\" (UID: \"3ae7814e-ecb2-4a33-91d9-023cc8269455\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn"
Sep 19 10:36:29.926689 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.926666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcxm6\" (UniqueName: \"kubernetes.io/projected/12123b50-d178-4a45-aab4-1785cba2616f-kube-api-access-dcxm6\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:29.926916 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.926899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-bound-sa-token\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.927155 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.926910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9rgk\" (UniqueName: \"kubernetes.io/projected/97db0dbe-bb49-4a08-99dc-0766b211bfbe-kube-api-access-b9rgk\") pod \"ingress-canary-mjqjg\" (UID: \"97db0dbe-bb49-4a08-99dc-0766b211bfbe\") " pod="openshift-ingress-canary/ingress-canary-mjqjg"
Sep 19 10:36:29.927155 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.926995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwgnw\" (UniqueName: \"kubernetes.io/projected/46872ad2-8593-4f10-8d29-defbae3b3c50-kube-api-access-pwgnw\") pod \"cluster-proxy-proxy-agent-7d4959c68f-bhfs7\" (UID: \"46872ad2-8593-4f10-8d29-defbae3b3c50\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:29.928600 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.928581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vszqz\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-kube-api-access-vszqz\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:29.993672 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:29.993629    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:36:30.015963 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:30.015928    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"
Sep 19 10:36:30.035807 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:30.035778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn"
Sep 19 10:36:30.076983 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:30.076957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mjqjg"
Sep 19 10:36:30.145318 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:30.145271    2580 generic.go:356] "Generic (PLEG): container finished" podID="b8d90df9-5745-4b48-bd08-b5cdb66c5c9d" containerID="11b3d3f7718252c76af05f6adc9e4377c623ebee52447429a866e87b14f6b796" exitCode=0
Sep 19 10:36:30.146214 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:30.145342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jvzl" event={"ID":"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d","Type":"ContainerDied","Data":"11b3d3f7718252c76af05f6adc9e4377c623ebee52447429a866e87b14f6b796"}
Sep 19 10:36:30.326277 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:30.326009    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"]
Sep 19 10:36:30.329531 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:36:30.329500    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494a2207_e136_4cc8_8e00_432f60322e67.slice/crio-d91eb5185e98c0919c01960fdc31a21748200a0e454a40fde5ebae138ac8d396 WatchSource:0}: Error finding container d91eb5185e98c0919c01960fdc31a21748200a0e454a40fde5ebae138ac8d396: Status 404 returned error can't find the container with id d91eb5185e98c0919c01960fdc31a21748200a0e454a40fde5ebae138ac8d396
Sep 19 10:36:30.333550 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:30.333520    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"]
Sep 19 10:36:30.336080 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:36:30.336051    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46872ad2_8593_4f10_8d29_defbae3b3c50.slice/crio-2b3deb85be115ba3fc0ce6fac99c16ae113dd466af448809caba993b5a72c6ef WatchSource:0}: Error finding container 2b3deb85be115ba3fc0ce6fac99c16ae113dd466af448809caba993b5a72c6ef: Status 404 returned error can't find the container with id 2b3deb85be115ba3fc0ce6fac99c16ae113dd466af448809caba993b5a72c6ef
Sep 19 10:36:30.347158 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:30.347122    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn"]
Sep 19 10:36:30.349799 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:36:30.349776    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ae7814e_ecb2_4a33_91d9_023cc8269455.slice/crio-e3d0cf5734f18ed8fa6145124f7300ef6eb52678d50d21aa6bfb12a7db402188 WatchSource:0}: Error finding container e3d0cf5734f18ed8fa6145124f7300ef6eb52678d50d21aa6bfb12a7db402188: Status 404 returned error can't find the container with id e3d0cf5734f18ed8fa6145124f7300ef6eb52678d50d21aa6bfb12a7db402188
Sep 19 10:36:30.388980 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:30.388947    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mjqjg"]
Sep 19 10:36:30.391992 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:36:30.391966    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97db0dbe_bb49_4a08_99dc_0766b211bfbe.slice/crio-72b85306d51d1f22e0ca654978db882f02636edf14a17b0176e9c538573c41f3 WatchSource:0}: Error finding container 72b85306d51d1f22e0ca654978db882f02636edf14a17b0176e9c538573c41f3: Status 404 returned error can't find the container with id 72b85306d51d1f22e0ca654978db882f02636edf14a17b0176e9c538573c41f3
Sep 19 10:36:30.422519 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:30.422487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:30.422652 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:30.422549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:30.422691 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:30.422646    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 10:36:30.422691 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:30.422658    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 10:36:30.422691 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:30.422666    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f679b797-cmc99: secret "image-registry-tls" not found
Sep 19 10:36:30.422813 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:30.422714    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls podName:12123b50-d178-4a45-aab4-1785cba2616f nodeName:}" failed. No retries permitted until 2026-09-19 10:36:31.42270144 +0000 UTC m=+36.081446592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls") pod "dns-default-ncb5v" (UID: "12123b50-d178-4a45-aab4-1785cba2616f") : secret "dns-default-metrics-tls" not found
Sep 19 10:36:30.422813 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:30.422727    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls podName:1181bcde-49d9-4582-9e19-c9603fc3c7cd nodeName:}" failed. No retries permitted until 2026-09-19 10:36:31.422720754 +0000 UTC m=+36.081465891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls") pod "image-registry-69f679b797-cmc99" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd") : secret "image-registry-tls" not found
Sep 19 10:36:31.152627 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:31.152586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn" event={"ID":"3ae7814e-ecb2-4a33-91d9-023cc8269455","Type":"ContainerStarted","Data":"e3d0cf5734f18ed8fa6145124f7300ef6eb52678d50d21aa6bfb12a7db402188"}
Sep 19 10:36:31.154455 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:31.154422    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm" event={"ID":"494a2207-e136-4cc8-8e00-432f60322e67","Type":"ContainerStarted","Data":"d91eb5185e98c0919c01960fdc31a21748200a0e454a40fde5ebae138ac8d396"}
Sep 19 10:36:31.156094 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:31.156059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7" event={"ID":"46872ad2-8593-4f10-8d29-defbae3b3c50","Type":"ContainerStarted","Data":"2b3deb85be115ba3fc0ce6fac99c16ae113dd466af448809caba993b5a72c6ef"}
Sep 19 10:36:31.167916 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:31.167874    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jvzl" event={"ID":"b8d90df9-5745-4b48-bd08-b5cdb66c5c9d","Type":"ContainerStarted","Data":"4f5a2451efd6f3f93c10601e7887e1e173064b136f77b643e6f22383eeeecd8f"}
Sep 19 10:36:31.178549 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:31.178406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mjqjg" event={"ID":"97db0dbe-bb49-4a08-99dc-0766b211bfbe","Type":"ContainerStarted","Data":"72b85306d51d1f22e0ca654978db882f02636edf14a17b0176e9c538573c41f3"}
Sep 19 10:36:31.196567 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:31.196500    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9jvzl" podStartSLOduration=7.064912038 podStartE2EDuration="35.196475959s" totalImagesPullingTime="28.131563921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:35:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:31.194841528 +0000 UTC m=+35.853586689" watchObservedRunningTime="2026-09-19 10:36:31.196475959 +0000 UTC m=+35.855221117"
Sep 19 10:36:31.250231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:31.250148    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:31.435513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:31.434320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:31.435513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:31.434420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:31.435513 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:31.434676    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 10:36:31.435513 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:31.434798    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls podName:12123b50-d178-4a45-aab4-1785cba2616f nodeName:}" failed. No retries permitted until 2026-09-19 10:36:33.434775386 +0000 UTC m=+38.093520543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls") pod "dns-default-ncb5v" (UID: "12123b50-d178-4a45-aab4-1785cba2616f") : secret "dns-default-metrics-tls" not found
Sep 19 10:36:31.435513 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:31.435271    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 10:36:31.435513 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:31.435286    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f679b797-cmc99: secret "image-registry-tls" not found
Sep 19 10:36:31.435513 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:31.435340    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls podName:1181bcde-49d9-4582-9e19-c9603fc3c7cd nodeName:}" failed. No retries permitted until 2026-09-19 10:36:33.435324796 +0000 UTC m=+38.094069948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls") pod "image-registry-69f679b797-cmc99" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd") : secret "image-registry-tls" not found
Sep 19 10:36:32.368222 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:32.368181    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:32.650215 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:32.650019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:32.699291 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:32.699193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:33.455010 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:33.454915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:33.455531 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:33.455073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:33.455531 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:33.455084    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 10:36:33.455531 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:33.455109    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f679b797-cmc99: secret "image-registry-tls" not found
Sep 19 10:36:33.455531 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:33.455179    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls podName:1181bcde-49d9-4582-9e19-c9603fc3c7cd nodeName:}" failed. No retries permitted until 2026-09-19 10:36:37.455156572 +0000 UTC m=+42.113901721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls") pod "image-registry-69f679b797-cmc99" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd") : secret "image-registry-tls" not found
Sep 19 10:36:33.455531 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:33.455187    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 10:36:33.455531 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:33.455261    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls podName:12123b50-d178-4a45-aab4-1785cba2616f nodeName:}" failed. No retries permitted until 2026-09-19 10:36:37.455243844 +0000 UTC m=+42.113988983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls") pod "dns-default-ncb5v" (UID: "12123b50-d178-4a45-aab4-1785cba2616f") : secret "dns-default-metrics-tls" not found
Sep 19 10:36:34.453454 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:34.453407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:34.556855 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:34.556813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:35.068346 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:35.068307    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:35.121897 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:35.121865    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:37.488173 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:37.488128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:37.488639 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:37.488202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:37.488639 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:37.488319    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 10:36:37.488639 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:37.488346    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f679b797-cmc99: secret "image-registry-tls" not found
Sep 19 10:36:37.488639 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:37.488384    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 10:36:37.488639 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:37.488419    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls podName:1181bcde-49d9-4582-9e19-c9603fc3c7cd nodeName:}" failed. No retries permitted until 2026-09-19 10:36:45.488396723 +0000 UTC m=+50.147141863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls") pod "image-registry-69f679b797-cmc99" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd") : secret "image-registry-tls" not found
Sep 19 10:36:37.488639 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:37.488444    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls podName:12123b50-d178-4a45-aab4-1785cba2616f nodeName:}" failed. No retries permitted until 2026-09-19 10:36:45.488428598 +0000 UTC m=+50.147173763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls") pod "dns-default-ncb5v" (UID: "12123b50-d178-4a45-aab4-1785cba2616f") : secret "dns-default-metrics-tls" not found
Sep 19 10:36:38.272418 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:38.272378    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:38.372321 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:38.372287    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:38.472217 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:38.472177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:38.533849 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:38.533744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:40.206507 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:40.206463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mjqjg" event={"ID":"97db0dbe-bb49-4a08-99dc-0766b211bfbe","Type":"ContainerStarted","Data":"ab2051f7a695ee109f8ecc7d356111ccad66ff0b8880aaff34aa55e9b0f602e4"}
Sep 19 10:36:40.208048 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:40.208026    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn" event={"ID":"3ae7814e-ecb2-4a33-91d9-023cc8269455","Type":"ContainerStarted","Data":"12567b6526c1e5532a3427292135da722503b956ad50d285915c93f1b18b6f08"}
Sep 19 10:36:40.209519 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:40.209486    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm" event={"ID":"494a2207-e136-4cc8-8e00-432f60322e67","Type":"ContainerStarted","Data":"47342d9a18d608191dca3ca083c6088196fb3efcf0ec1b86c0775d03f1ea9914"}
Sep 19 10:36:40.209669 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:40.209650    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"
Sep 19 10:36:40.211637 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:40.211614    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm"
Sep 19 10:36:40.211743 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:40.211683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7" event={"ID":"46872ad2-8593-4f10-8d29-defbae3b3c50","Type":"ContainerStarted","Data":"810be5ba67003e8d02d05ec1f87bb1cfc64d346ebccb14c71eb16ca072175de9"}
Sep 19 10:36:40.211743 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:40.211713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7" event={"ID":"46872ad2-8593-4f10-8d29-defbae3b3c50","Type":"ContainerStarted","Data":"a5c337b84cd524c6c6707e66423874ac05697796060c9d0dedc52a8c2eb3374b"}
Sep 19 10:36:40.211743 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:40.211728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7" event={"ID":"46872ad2-8593-4f10-8d29-defbae3b3c50","Type":"ContainerStarted","Data":"5bf25e26f4d87a70b5f37cc3ecd0594c747b978be81ad757de258b9b25f9f522"}
Sep 19 10:36:40.224695 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:40.224637    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mjqjg" podStartSLOduration=2.517761837 podStartE2EDuration="11.224614314s" totalImagesPullingTime="8.706852477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:40.223123706 +0000 UTC m=+44.881868859" watchObservedRunningTime="2026-09-19 10:36:40.224614314 +0000 UTC m=+44.883359477"
Sep 19 10:36:40.241140 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:40.241077    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7" podStartSLOduration=18.454755531 podStartE2EDuration="27.241055441s" totalImagesPullingTime="8.78629991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:40.240130925 +0000 UTC m=+44.898876085" watchObservedRunningTime="2026-09-19 10:36:40.241055441 +0000 UTC m=+44.899800602"
Sep 19 10:36:40.254211 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:40.254154    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b4775464-57vzm" podStartSLOduration=18.446130875 podStartE2EDuration="27.254138276s" totalImagesPullingTime="8.808007401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:40.253804759 +0000 UTC m=+44.912549920" watchObservedRunningTime="2026-09-19 10:36:40.254138276 +0000 UTC m=+44.912883437"
Sep 19 10:36:40.267485 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:40.267427    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76967986bd-krjfn" podStartSLOduration=18.494785154 podStartE2EDuration="27.267407925s" totalImagesPullingTime="8.772622771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:40.266723317 +0000 UTC m=+44.925468477" watchObservedRunningTime="2026-09-19 10:36:40.267407925 +0000 UTC m=+44.926153085"
Sep 19 10:36:45.548449 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:45.548399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:36:45.548990 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:45.548488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:36:45.548990 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:45.548569    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 10:36:45.548990 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:45.548596    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 10:36:45.548990 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:45.548608    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f679b797-cmc99: secret "image-registry-tls" not found
Sep 19 10:36:45.548990 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:45.548660    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls podName:1181bcde-49d9-4582-9e19-c9603fc3c7cd nodeName:}" failed. No retries permitted until 2026-09-19 10:37:01.548642755 +0000 UTC m=+66.207387913 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls") pod "image-registry-69f679b797-cmc99" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd") : secret "image-registry-tls" not found
Sep 19 10:36:45.548990 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:36:45.548674    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls podName:12123b50-d178-4a45-aab4-1785cba2616f nodeName:}" failed. No retries permitted until 2026-09-19 10:37:01.548667399 +0000 UTC m=+66.207412537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls") pod "dns-default-ncb5v" (UID: "12123b50-d178-4a45-aab4-1785cba2616f") : secret "dns-default-metrics-tls" not found
Sep 19 10:36:53.402438 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:53.402402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:53.452664 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:53.452630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:54.128444 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:54.128417    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kx7wh"
Sep 19 10:36:54.266483 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:54.266450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:54.394628 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:54.394544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:54.446709 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:54.446676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:54.746908 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:54.746873    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:54.801537 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:54.801508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:57.318703 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:57.318663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:59.479271 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:59.479222    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:59.579530 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:59.579468    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:36:59.678888 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:36:59.678851    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:00.487977 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:00.487944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:00.558093 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:00.558067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:00.611366 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:00.611337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:01.566527 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.566487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:37:01.566938 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.566550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:37:01.566938 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:01.566646    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 10:37:01.566938 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:01.566656    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 10:37:01.566938 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:01.566667    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f679b797-cmc99: secret "image-registry-tls" not found
Sep 19 10:37:01.566938 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:01.566714    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls podName:12123b50-d178-4a45-aab4-1785cba2616f nodeName:}" failed. No retries permitted until 2026-09-19 10:37:33.566699984 +0000 UTC m=+98.225445135 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls") pod "dns-default-ncb5v" (UID: "12123b50-d178-4a45-aab4-1785cba2616f") : secret "dns-default-metrics-tls" not found
Sep 19 10:37:01.566938 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:01.566726    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls podName:1181bcde-49d9-4582-9e19-c9603fc3c7cd nodeName:}" failed. No retries permitted until 2026-09-19 10:37:33.566720227 +0000 UTC m=+98.225465365 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls") pod "image-registry-69f679b797-cmc99" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd") : secret "image-registry-tls" not found
Sep 19 10:37:01.666861 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.666827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:37:01.667012 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.666887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:37:01.669030 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.669004    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 10:37:01.669143 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.669129    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 10:37:01.677825 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:01.677797    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 19 10:37:01.677929 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:01.677879    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs podName:162318c8-6ce5-4dd0-9f6c-6c61d1858816 nodeName:}" failed. No retries permitted until 2026-09-19 10:38:05.677862802 +0000 UTC m=+130.336607940 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs") pod "network-metrics-daemon-fhc4f" (UID: "162318c8-6ce5-4dd0-9f6c-6c61d1858816") : secret "metrics-daemon-secret" not found
Sep 19 10:37:01.681194 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.681165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f24a9314-df59-4a55-8d61-9b6f278c322d-original-pull-secret\") pod \"global-pull-secret-syncer-wxp6f\" (UID: \"f24a9314-df59-4a55-8d61-9b6f278c322d\") " pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:37:01.767724 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.767676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmns2\" (UniqueName: \"kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2\") pod \"network-check-target-hz8mn\" (UID: \"291c0f79-1f63-4e4c-bcf3-6f85ee2edee6\") " pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:37:01.769828 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.769810    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 10:37:01.780275 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.780253    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 10:37:01.791381 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.791344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmns2\" (UniqueName: \"kubernetes.io/projected/291c0f79-1f63-4e4c-bcf3-6f85ee2edee6-kube-api-access-vmns2\") pod \"network-check-target-hz8mn\" (UID: \"291c0f79-1f63-4e4c-bcf3-6f85ee2edee6\") " pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:37:01.960170 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.960135    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxp6f"
Sep 19 10:37:01.967455 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.967431    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ql7r7\""
Sep 19 10:37:01.976326 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:01.976305    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:37:02.228997 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:02.228903    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wxp6f"]
Sep 19 10:37:02.233217 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:37:02.233185    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24a9314_df59_4a55_8d61_9b6f278c322d.slice/crio-8d5e1b190a9973a2af28bb0517a995265db4142252799c746d5dfa7dae95d30e WatchSource:0}: Error finding container 8d5e1b190a9973a2af28bb0517a995265db4142252799c746d5dfa7dae95d30e: Status 404 returned error can't find the container with id 8d5e1b190a9973a2af28bb0517a995265db4142252799c746d5dfa7dae95d30e
Sep 19 10:37:02.251952 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:02.251925    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hz8mn"]
Sep 19 10:37:02.254854 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:37:02.254827    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291c0f79_1f63_4e4c_bcf3_6f85ee2edee6.slice/crio-9527743543b86a72ad37e91cd66a53057108585b8ab562e68d2db1800c5b2de3 WatchSource:0}: Error finding container 9527743543b86a72ad37e91cd66a53057108585b8ab562e68d2db1800c5b2de3: Status 404 returned error can't find the container with id 9527743543b86a72ad37e91cd66a53057108585b8ab562e68d2db1800c5b2de3
Sep 19 10:37:02.278167 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:02.278133    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wxp6f" event={"ID":"f24a9314-df59-4a55-8d61-9b6f278c322d","Type":"ContainerStarted","Data":"8d5e1b190a9973a2af28bb0517a995265db4142252799c746d5dfa7dae95d30e"}
Sep 19 10:37:02.279243 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:02.279222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hz8mn" event={"ID":"291c0f79-1f63-4e4c-bcf3-6f85ee2edee6","Type":"ContainerStarted","Data":"9527743543b86a72ad37e91cd66a53057108585b8ab562e68d2db1800c5b2de3"}
Sep 19 10:37:03.427952 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:03.427854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:05.408315 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:05.408277    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:05.466654 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:05.466617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:06.904912 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:06.904872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:06.960047 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:06.960004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:08.299073 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:08.299043    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wxp6f" event={"ID":"f24a9314-df59-4a55-8d61-9b6f278c322d","Type":"ContainerStarted","Data":"d8ebee2133d482a36eedb2e348269c825fda37344cd284376ffbce37bcfbcbeb"}
Sep 19 10:37:08.300631 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:08.300605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hz8mn" event={"ID":"291c0f79-1f63-4e4c-bcf3-6f85ee2edee6","Type":"ContainerStarted","Data":"2e6122d2ae05ca5b17c8df7a77815674452eca967756fdacb245372edd426220"}
Sep 19 10:37:08.300776 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:08.300658    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:37:08.313193 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:08.313152    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wxp6f" podStartSLOduration=67.272033626 podStartE2EDuration="1m12.313137575s" totalImagesPullingTime="5.041103949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:37:08.312523178 +0000 UTC m=+72.971268338" watchObservedRunningTime="2026-09-19 10:37:08.313137575 +0000 UTC m=+72.971882734"
Sep 19 10:37:08.324920 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:08.324844    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hz8mn" podStartSLOduration=67.317653266 podStartE2EDuration="1m12.324828314s" totalImagesPullingTime="5.007175048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:37:08.324637458 +0000 UTC m=+72.983382617" watchObservedRunningTime="2026-09-19 10:37:08.324828314 +0000 UTC m=+72.983573468"
Sep 19 10:37:09.494538 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:09.494505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:09.593978 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:09.593942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:10.453422 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:10.453388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:10.686667 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:10.686640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:10.788128 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:10.788092    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:11.528034 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:11.528000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:11.582649 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:11.582614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:15.799105 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.799071    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fg4gh"]
Sep 19 10:37:15.802048 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.802033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:15.803862 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.803840    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 19 10:37:15.804171 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.804150    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5c6t4\""
Sep 19 10:37:15.804336 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.804299    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:15.804738 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.804723    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 19 10:37:15.804738 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.804730    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 19 10:37:15.804915 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.804744    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 19 10:37:15.810829 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.810808    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fg4gh"]
Sep 19 10:37:15.858105 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.858065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:15.877194 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.877160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:15.877413 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.877203    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w46z\" (UniqueName: \"kubernetes.io/projected/16129530-3326-4f01-8f22-b9ea97538fd9-kube-api-access-8w46z\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:15.877413 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.877223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:15.877413 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.877276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/16129530-3326-4f01-8f22-b9ea97538fd9-crio-socket\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:15.877413 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.877406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/16129530-3326-4f01-8f22-b9ea97538fd9-data-volume\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:15.977843 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.977803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/16129530-3326-4f01-8f22-b9ea97538fd9-crio-socket\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:15.978028 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.977897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/16129530-3326-4f01-8f22-b9ea97538fd9-data-volume\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:15.978028 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.977926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/16129530-3326-4f01-8f22-b9ea97538fd9-crio-socket\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:15.978028 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.977938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:15.978028 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.977984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w46z\" (UniqueName: \"kubernetes.io/projected/16129530-3326-4f01-8f22-b9ea97538fd9-kube-api-access-8w46z\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:15.978028 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.978004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:15.978028 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:15.978027    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 10:37:15.978306 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:15.978062    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 10:37:15.978306 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:15.978089    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls podName:16129530-3326-4f01-8f22-b9ea97538fd9 nodeName:}" failed. No retries permitted until 2026-09-19 10:37:16.478066362 +0000 UTC m=+81.136811502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fg4gh" (UID: "16129530-3326-4f01-8f22-b9ea97538fd9") : secret "insights-runtime-extractor-tls" not found
Sep 19 10:37:15.978306 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:15.978119    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm podName:16129530-3326-4f01-8f22-b9ea97538fd9 nodeName:}" failed. No retries permitted until 2026-09-19 10:37:16.478103294 +0000 UTC m=+81.136848431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fg4gh" (UID: "16129530-3326-4f01-8f22-b9ea97538fd9") : configmap "kube-rbac-proxy" not found
Sep 19 10:37:15.978306 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.978250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/16129530-3326-4f01-8f22-b9ea97538fd9-data-volume\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:15.985974 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:15.985951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w46z\" (UniqueName: \"kubernetes.io/projected/16129530-3326-4f01-8f22-b9ea97538fd9-kube-api-access-8w46z\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:16.200684 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:16.200643    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:16.254875 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:16.254841    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:16.308663 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:16.308635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:16.482675 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:16.482579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:16.482675 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:16.482626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:16.482893 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:16.482741    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 10:37:16.482893 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:16.482791    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 10:37:16.482893 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:16.482829    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm podName:16129530-3326-4f01-8f22-b9ea97538fd9 nodeName:}" failed. No retries permitted until 2026-09-19 10:37:17.482811362 +0000 UTC m=+82.141556502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fg4gh" (UID: "16129530-3326-4f01-8f22-b9ea97538fd9") : configmap "kube-rbac-proxy" not found
Sep 19 10:37:16.482893 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:16.482857    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls podName:16129530-3326-4f01-8f22-b9ea97538fd9 nodeName:}" failed. No retries permitted until 2026-09-19 10:37:17.482842287 +0000 UTC m=+82.141587443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fg4gh" (UID: "16129530-3326-4f01-8f22-b9ea97538fd9") : secret "insights-runtime-extractor-tls" not found
Sep 19 10:37:16.571148 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:16.571113    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:16.667565 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:16.667521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:16.767697 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:16.767545    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:16.911382 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:16.911330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:17.012459 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:17.012422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:17.491031 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:17.490995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:17.491235 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:17.491053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:17.491235 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:17.491131    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 10:37:17.491235 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:17.491144    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 10:37:17.491235 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:17.491195    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls podName:16129530-3326-4f01-8f22-b9ea97538fd9 nodeName:}" failed. No retries permitted until 2026-09-19 10:37:19.491181026 +0000 UTC m=+84.149926164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fg4gh" (UID: "16129530-3326-4f01-8f22-b9ea97538fd9") : secret "insights-runtime-extractor-tls" not found
Sep 19 10:37:17.491235 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:17.491211    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm podName:16129530-3326-4f01-8f22-b9ea97538fd9 nodeName:}" failed. No retries permitted until 2026-09-19 10:37:19.491202864 +0000 UTC m=+84.149948002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fg4gh" (UID: "16129530-3326-4f01-8f22-b9ea97538fd9") : configmap "kube-rbac-proxy" not found
Sep 19 10:37:17.823951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:17.823871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:17.924723 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:17.924679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:17.961506 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:17.961482    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dkccw_4c1b1206-2361-42b9-b787-b49f81c893c7/dns-node-resolver/0.log"
Sep 19 10:37:18.024955 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:18.024921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:18.545263 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:18.545230    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:18.560705 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:18.560678    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-49h5q_b5bae54d-ce02-46ec-a38c-c43f61441d7d/node-ca/0.log"
Sep 19 10:37:19.398347 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:19.398313    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:19.506803 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:19.506720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:19.506950 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:19.506818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:19.506950 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:19.506882    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 10:37:19.506950 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:19.506942    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 10:37:19.507071 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:19.506949    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls podName:16129530-3326-4f01-8f22-b9ea97538fd9 nodeName:}" failed. No retries permitted until 2026-09-19 10:37:23.506934606 +0000 UTC m=+88.165679743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fg4gh" (UID: "16129530-3326-4f01-8f22-b9ea97538fd9") : secret "insights-runtime-extractor-tls" not found
Sep 19 10:37:19.507071 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:19.507003    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm podName:16129530-3326-4f01-8f22-b9ea97538fd9 nodeName:}" failed. No retries permitted until 2026-09-19 10:37:23.506989503 +0000 UTC m=+88.165734643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fg4gh" (UID: "16129530-3326-4f01-8f22-b9ea97538fd9") : configmap "kube-rbac-proxy" not found
Sep 19 10:37:19.760710 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:19.760683    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mjqjg_97db0dbe-bb49-4a08-99dc-0766b211bfbe/serve-healthcheck-canary/0.log"
Sep 19 10:37:22.180368 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:22.180337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:22.617280 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:22.617237    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:22.721389 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:22.721354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:22.818522 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:22.818495    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:23.058539 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:23.058504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:23.159703 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:23.159670    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:23.352589 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:23.352489    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:23.539822 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:23.539786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:23.540056 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:23.539835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:23.540056 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:23.539922    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 10:37:23.540056 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:23.539950    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 10:37:23.540056 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:23.540007    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls podName:16129530-3326-4f01-8f22-b9ea97538fd9 nodeName:}" failed. No retries permitted until 2026-09-19 10:37:31.539985819 +0000 UTC m=+96.198730957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fg4gh" (UID: "16129530-3326-4f01-8f22-b9ea97538fd9") : secret "insights-runtime-extractor-tls" not found
Sep 19 10:37:23.540056 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:23.540026    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm podName:16129530-3326-4f01-8f22-b9ea97538fd9 nodeName:}" failed. No retries permitted until 2026-09-19 10:37:31.5400168 +0000 UTC m=+96.198761937 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fg4gh" (UID: "16129530-3326-4f01-8f22-b9ea97538fd9") : configmap "kube-rbac-proxy" not found
Sep 19 10:37:23.713105 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:23.713074    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:24.712566 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:24.712531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:25.116711 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:25.116620    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:25.316041 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:25.315995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:25.517490 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:25.517450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:25.569890 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:25.569854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:25.915811 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:25.915708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:26.427788 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:26.427741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:26.527184 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:26.527146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:26.712846 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:26.712812    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:26.910970 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:26.910934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:27.314060 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:27.314023    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:27.365187 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:27.365149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:27.511817 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:27.511783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:27.914303 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:27.914260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:28.515049 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:28.514996    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:28.913616 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:28.913523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:29.014645 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:29.014605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:29.115700 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:29.115669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:29.315583 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:29.315550    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:29.511904 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:29.511863    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:29.977770 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:29.977716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:30.315974 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:30.315882    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:30.909395 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:30.909365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:31.309631 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:31.309601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:31.606939 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:31.606846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:31.606939 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:31.606895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:31.607138 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:31.607021    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 10:37:31.607138 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:37:31.607106    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls podName:16129530-3326-4f01-8f22-b9ea97538fd9 nodeName:}" failed. No retries permitted until 2026-09-19 10:37:47.60708689 +0000 UTC m=+112.265832052 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fg4gh" (UID: "16129530-3326-4f01-8f22-b9ea97538fd9") : secret "insights-runtime-extractor-tls" not found
Sep 19 10:37:31.607422 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:31.607406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/16129530-3326-4f01-8f22-b9ea97538fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:31.712673 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:31.712641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:31.813639 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:31.813609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:31.865831 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:31.865678    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:32.016196 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:32.016164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:32.118049 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:32.117962    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:32.313601 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:32.313569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:32.414168 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:32.414069    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:32.716518 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:32.716485    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:32.824982 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:32.824951    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:32.881044 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:32.881011    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:33.017846 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.015177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:33.066740 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.066705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:33.123335 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.123303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:33.223846 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.223813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:33.323548 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.323506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:33.607963 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.607884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:33.623633 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.623604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:37:33.623725 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.623651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:37:33.626192 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.626170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12123b50-d178-4a45-aab4-1785cba2616f-metrics-tls\") pod \"dns-default-ncb5v\" (UID: \"12123b50-d178-4a45-aab4-1785cba2616f\") " pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:37:33.626310 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.626217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls\") pod \"image-registry-69f679b797-cmc99\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") " pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:37:33.673227 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.673191    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qlvz2\""
Sep 19 10:37:33.681630 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.681609    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:37:33.800544 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.800513    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:33.904433 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.904349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lp9gc\""
Sep 19 10:37:33.913396 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.913376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:37:33.956461 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:33.956433    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ncb5v"]
Sep 19 10:37:33.959309 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:37:33.959264    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12123b50_d178_4a45_aab4_1785cba2616f.slice/crio-4fb5c215caa0ad71f6f652bd862cb01d28f6eeffa7746ff3e0c24fc8e5e82ee5 WatchSource:0}: Error finding container 4fb5c215caa0ad71f6f652bd862cb01d28f6eeffa7746ff3e0c24fc8e5e82ee5: Status 404 returned error can't find the container with id 4fb5c215caa0ad71f6f652bd862cb01d28f6eeffa7746ff3e0c24fc8e5e82ee5
Sep 19 10:37:34.107215 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:34.107178    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:34.184379 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:34.184345    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69f679b797-cmc99"]
Sep 19 10:37:34.187125 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:37:34.187101    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1181bcde_49d9_4582_9e19_c9603fc3c7cd.slice/crio-daee7568632c64af5a4538453c7df322c853c1ef1ecbd21a401a8f5093e0e084 WatchSource:0}: Error finding container daee7568632c64af5a4538453c7df322c853c1ef1ecbd21a401a8f5093e0e084: Status 404 returned error can't find the container with id daee7568632c64af5a4538453c7df322c853c1ef1ecbd21a401a8f5093e0e084
Sep 19 10:37:34.380705 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:34.380671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f679b797-cmc99" event={"ID":"1181bcde-49d9-4582-9e19-c9603fc3c7cd","Type":"ContainerStarted","Data":"5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc"}
Sep 19 10:37:34.381192 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:34.380729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f679b797-cmc99" event={"ID":"1181bcde-49d9-4582-9e19-c9603fc3c7cd","Type":"ContainerStarted","Data":"daee7568632c64af5a4538453c7df322c853c1ef1ecbd21a401a8f5093e0e084"}
Sep 19 10:37:34.381192 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:34.380799    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:37:34.382035 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:34.382005    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ncb5v" event={"ID":"12123b50-d178-4a45-aab4-1785cba2616f","Type":"ContainerStarted","Data":"4fb5c215caa0ad71f6f652bd862cb01d28f6eeffa7746ff3e0c24fc8e5e82ee5"}
Sep 19 10:37:34.399052 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:34.398990    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69f679b797-cmc99" podStartSLOduration=77.398969219 podStartE2EDuration="1m17.398969219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:37:34.398163537 +0000 UTC m=+99.056908697" watchObservedRunningTime="2026-09-19 10:37:34.398969219 +0000 UTC m=+99.057714380"
Sep 19 10:37:34.446702 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:34.446608    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:34.546263 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:34.546234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:35.023204 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:35.022983    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:35.072138 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:35.072095    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:36.390317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:36.390276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ncb5v" event={"ID":"12123b50-d178-4a45-aab4-1785cba2616f","Type":"ContainerStarted","Data":"006d69e3e9168edd41c7ca190592d35d3bfcda386e257d1112b0520862c04666"}
Sep 19 10:37:36.390717 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:36.390335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ncb5v" event={"ID":"12123b50-d178-4a45-aab4-1785cba2616f","Type":"ContainerStarted","Data":"7555091a5666b0fc0afddd0334d9691657baa527781809d4d5637076e10cf6c9"}
Sep 19 10:37:36.390717 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:36.390376    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:37:36.406639 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:36.406544    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ncb5v" podStartSLOduration=65.852365424 podStartE2EDuration="1m7.406523753s" totalImagesPullingTime="1.554158329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:37:36.405111833 +0000 UTC m=+101.063856993" watchObservedRunningTime="2026-09-19 10:37:36.406523753 +0000 UTC m=+101.065268916"
Sep 19 10:37:39.306672 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:39.306643    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hz8mn"
Sep 19 10:37:41.596402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:41.596370    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:41.704380 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:41.704345    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:41.795870 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:41.795828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:41.896330 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:41.896230    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:43.277219 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:43.277186    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:43.377907 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:43.377868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:43.449700 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:43.449654    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:44.352089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:44.352049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:44.624856 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:44.624772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:44.723933 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:44.723894    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:44.773808 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:44.773775    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:46.323642 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:46.323602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:46.382561 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:46.382531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:46.396195 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:46.396169    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ncb5v"
Sep 19 10:37:47.641905 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:47.641860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:47.644400 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:47.644381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/16129530-3326-4f01-8f22-b9ea97538fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg4gh\" (UID: \"16129530-3326-4f01-8f22-b9ea97538fd9\") " pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:47.911800 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:47.911666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fg4gh"
Sep 19 10:37:48.178116 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:48.178032    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fg4gh"]
Sep 19 10:37:48.181171 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:37:48.181141    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16129530_3326_4f01_8f22_b9ea97538fd9.slice/crio-0278c71f920c11b391456ceb139d169e5ea9093899338172e61c68adaca48650 WatchSource:0}: Error finding container 0278c71f920c11b391456ceb139d169e5ea9093899338172e61c68adaca48650: Status 404 returned error can't find the container with id 0278c71f920c11b391456ceb139d169e5ea9093899338172e61c68adaca48650
Sep 19 10:37:48.432044 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:48.432013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fg4gh" event={"ID":"16129530-3326-4f01-8f22-b9ea97538fd9","Type":"ContainerStarted","Data":"b62eec6757ba45c39f400386b02f29eed2933b51ee9255bfa3f631891af7ff2e"}
Sep 19 10:37:48.432214 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:48.432093    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fg4gh" event={"ID":"16129530-3326-4f01-8f22-b9ea97538fd9","Type":"ContainerStarted","Data":"0278c71f920c11b391456ceb139d169e5ea9093899338172e61c68adaca48650"}
Sep 19 10:37:49.022726 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:49.022686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:49.076208 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:49.076165    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:49.437414 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:49.437381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fg4gh" event={"ID":"16129530-3326-4f01-8f22-b9ea97538fd9","Type":"ContainerStarted","Data":"4a5e34c38585b8dfe876b7b292d7279651ecdfdca7be5fdd9e3a6aab3ec0b3ba"}
Sep 19 10:37:49.740175 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:49.740083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:49.842051 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:49.842016    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:49.995309 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:49.995194    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7" podUID="46872ad2-8593-4f10-8d29-defbae3b3c50" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 19 10:37:50.498889 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:50.498852    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:50.575017 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:50.574981    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:50.630822 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:50.630788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:51.028772 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:51.028714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:51.129454 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:51.129421    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:51.446530 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:51.446495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fg4gh" event={"ID":"16129530-3326-4f01-8f22-b9ea97538fd9","Type":"ContainerStarted","Data":"d6d952bb276c1d132bff0d070d9ea36b0ab9e672e13cd6b500355f3b410689f6"}
Sep 19 10:37:51.464900 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:51.464846    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fg4gh" podStartSLOduration=34.35341813 podStartE2EDuration="36.464829634s" totalImagesPullingTime="2.111411504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:37:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:37:51.463792803 +0000 UTC m=+116.122537962" watchObservedRunningTime="2026-09-19 10:37:51.464829634 +0000 UTC m=+116.123574793"
Sep 19 10:37:55.390452 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:55.390421    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:37:55.394803 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:55.394768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:55.493274 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:55.493238    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:55.596428 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:55.596398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:55.695032 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:55.695000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:57.629225 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:57.629190    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:57.729184 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:57.729146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:59.538081 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:59.538052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:59.638823 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:59.638786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:59.825459 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:59.825346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:59.933032 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:59.932986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:37:59.995536 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:37:59.995498    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7" podUID="46872ad2-8593-4f10-8d29-defbae3b3c50" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 19 10:38:02.252350 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:02.252308    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:02.353039 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:02.352987    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:03.552725 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:03.552684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:03.590993 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:03.590965    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69f679b797-cmc99"]
Sep 19 10:38:03.674160 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:03.674129    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:03.774626 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:03.774565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:04.443661 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:04.443621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:04.494351 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:04.494322    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:05.630423 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.630383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:05.773502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.773453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:38:05.776016 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.775986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162318c8-6ce5-4dd0-9f6c-6c61d1858816-metrics-certs\") pod \"network-metrics-daemon-fhc4f\" (UID: \"162318c8-6ce5-4dd0-9f6c-6c61d1858816\") " pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:38:05.844459 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.844420    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:05.871695 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.871666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6z24q\""
Sep 19 10:38:05.880705 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.880624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fhc4f"
Sep 19 10:38:05.890923 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.890887    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6kpzj"]
Sep 19 10:38:05.894317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.894293    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:05.896638 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.896601    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 19 10:38:05.896846 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.896822    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 19 10:38:05.896956 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.896940    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c42jv\""
Sep 19 10:38:05.897443 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.897425    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 19 10:38:05.897524 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.897445    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 19 10:38:05.897524 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.897481    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 19 10:38:05.897524 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.897445    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 19 10:38:05.933221 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.933183    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:05.975296 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.975253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:05.975296 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.975293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-wtmp\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:05.975509 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.975313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-accelerators-collector-config\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:05.975509 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.975382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-textfile\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:05.975509 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.975445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1f7e08f-bcc2-4995-b381-0efaf8682d67-sys\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:05.975509 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.975466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f7e08f-bcc2-4995-b381-0efaf8682d67-metrics-client-ca\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:05.975509 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.975480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6sp7\" (UniqueName: \"kubernetes.io/projected/f1f7e08f-bcc2-4995-b381-0efaf8682d67-kube-api-access-q6sp7\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:05.975509 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.975500    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-tls\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:05.975694 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:05.975534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1f7e08f-bcc2-4995-b381-0efaf8682d67-root\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.026474 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.026436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:06.076557 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.076520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1f7e08f-bcc2-4995-b381-0efaf8682d67-sys\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.076557 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.076560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f7e08f-bcc2-4995-b381-0efaf8682d67-metrics-client-ca\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.076809 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.076576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6sp7\" (UniqueName: \"kubernetes.io/projected/f1f7e08f-bcc2-4995-b381-0efaf8682d67-kube-api-access-q6sp7\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.076809 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.076596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-tls\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.076809 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.076623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1f7e08f-bcc2-4995-b381-0efaf8682d67-root\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.076809 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.076651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1f7e08f-bcc2-4995-b381-0efaf8682d67-sys\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.076809 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.076672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.076809 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.076719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-wtmp\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.076809 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.076729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1f7e08f-bcc2-4995-b381-0efaf8682d67-root\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.076809 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:38:06.076773    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 19 10:38:06.076809 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.076747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-accelerators-collector-config\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.077164 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:38:06.076852    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-tls podName:f1f7e08f-bcc2-4995-b381-0efaf8682d67 nodeName:}" failed. No retries permitted until 2026-09-19 10:38:06.576829121 +0000 UTC m=+131.235574276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-tls") pod "node-exporter-6kpzj" (UID: "f1f7e08f-bcc2-4995-b381-0efaf8682d67") : secret "node-exporter-tls" not found
Sep 19 10:38:06.077164 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.076871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-textfile\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.077164 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.076891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-wtmp\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.077268 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.077179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-textfile\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.077306 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.077265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f7e08f-bcc2-4995-b381-0efaf8682d67-metrics-client-ca\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.077363 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.077347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-accelerators-collector-config\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.079190 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.079170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.084274 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.084253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6sp7\" (UniqueName: \"kubernetes.io/projected/f1f7e08f-bcc2-4995-b381-0efaf8682d67-kube-api-access-q6sp7\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.126485 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.126445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:06.151515 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.151416    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fhc4f"]
Sep 19 10:38:06.154653 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:38:06.154627    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162318c8_6ce5_4dd0_9f6c_6c61d1858816.slice/crio-83f37ac9e09eeee74db8c2102a8da05e5c62416f3c5e1d7d04aab173627fe0bf WatchSource:0}: Error finding container 83f37ac9e09eeee74db8c2102a8da05e5c62416f3c5e1d7d04aab173627fe0bf: Status 404 returned error can't find the container with id 83f37ac9e09eeee74db8c2102a8da05e5c62416f3c5e1d7d04aab173627fe0bf
Sep 19 10:38:06.227502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.227456    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:06.327419 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.327381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:06.427529 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.427427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:06.497121 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.497084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fhc4f" event={"ID":"162318c8-6ce5-4dd0-9f6c-6c61d1858816","Type":"ContainerStarted","Data":"83f37ac9e09eeee74db8c2102a8da05e5c62416f3c5e1d7d04aab173627fe0bf"}
Sep 19 10:38:06.527358 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.527324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:06.580101 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.580073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-tls\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.582557 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.582529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1f7e08f-bcc2-4995-b381-0efaf8682d67-node-exporter-tls\") pod \"node-exporter-6kpzj\" (UID: \"f1f7e08f-bcc2-4995-b381-0efaf8682d67\") " pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.807875 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.807838    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6kpzj"
Sep 19 10:38:06.818455 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:38:06.818423    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f7e08f_bcc2_4995_b381_0efaf8682d67.slice/crio-dd12a0ee9855675d3bd3fb3af949eebec4000837aecfba31d0b821f5ee034a3e WatchSource:0}: Error finding container dd12a0ee9855675d3bd3fb3af949eebec4000837aecfba31d0b821f5ee034a3e: Status 404 returned error can't find the container with id dd12a0ee9855675d3bd3fb3af949eebec4000837aecfba31d0b821f5ee034a3e
Sep 19 10:38:06.891109 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.891076    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:06.982118 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:06.982068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:07.197501 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:07.197469    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:07.503253 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:07.503207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fhc4f" event={"ID":"162318c8-6ce5-4dd0-9f6c-6c61d1858816","Type":"ContainerStarted","Data":"892586c00ae7e234b7632d2b2fab96f2c9059836e6dd3be57e9f4cbe9ef053d9"}
Sep 19 10:38:07.503467 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:07.503269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fhc4f" event={"ID":"162318c8-6ce5-4dd0-9f6c-6c61d1858816","Type":"ContainerStarted","Data":"654d9f6a09267fcab527ffe3f63f5e5f8878c72e416f28a8c223026c906b8d56"}
Sep 19 10:38:07.504583 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:07.504556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6kpzj" event={"ID":"f1f7e08f-bcc2-4995-b381-0efaf8682d67","Type":"ContainerStarted","Data":"dd12a0ee9855675d3bd3fb3af949eebec4000837aecfba31d0b821f5ee034a3e"}
Sep 19 10:38:07.520174 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:07.520111    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fhc4f" podStartSLOduration=130.550593869 podStartE2EDuration="2m11.52008671s" totalImagesPullingTime="969.492841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:38:07.518530946 +0000 UTC m=+132.177276105" watchObservedRunningTime="2026-09-19 10:38:07.52008671 +0000 UTC m=+132.178831871"
Sep 19 10:38:07.600111 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:07.600076    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:07.700548 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:07.700519    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:08.509134 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:08.509099    2580 generic.go:356] "Generic (PLEG): container finished" podID="f1f7e08f-bcc2-4995-b381-0efaf8682d67" containerID="31d1d0b6da9a143126adf7eac681d30b3fa3613730b049865cdb25f92a73d348" exitCode=0
Sep 19 10:38:08.509570 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:08.509187    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6kpzj" event={"ID":"f1f7e08f-bcc2-4995-b381-0efaf8682d67","Type":"ContainerDied","Data":"31d1d0b6da9a143126adf7eac681d30b3fa3613730b049865cdb25f92a73d348"}
Sep 19 10:38:08.775863 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:08.775774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:08.841891 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:08.841860    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:08.934979 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:08.934938    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:09.211568 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:09.211532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:09.515058 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:09.514959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6kpzj" event={"ID":"f1f7e08f-bcc2-4995-b381-0efaf8682d67","Type":"ContainerStarted","Data":"777842c4c34e7bd3d81d03b85dbee2fa4c3f45fe07508b38d0b90ad92409ccae"}
Sep 19 10:38:09.515058 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:09.515008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6kpzj" event={"ID":"f1f7e08f-bcc2-4995-b381-0efaf8682d67","Type":"ContainerStarted","Data":"f057af9464d1b1663a140c5debd77637e8bed660b1d9a6a0b4911edd72fe3e75"}
Sep 19 10:38:09.535470 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:09.535391    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6kpzj" podStartSLOduration=3.457956844 podStartE2EDuration="4.535369396s" totalImagesPullingTime="1.007859605s" totalInitContainerRuntime="69.552947ms" isStatefulPod=true podCreationTimestamp="2026-09-19 10:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:38:09.531988322 +0000 UTC m=+134.190733486" watchObservedRunningTime="2026-09-19 10:38:09.535369396 +0000 UTC m=+134.194114558"
Sep 19 10:38:09.995818 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:09.995776    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7" podUID="46872ad2-8593-4f10-8d29-defbae3b3c50" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 19 10:38:09.995979 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:09.995845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7"
Sep 19 10:38:09.996325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:09.996288    2580 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"810be5ba67003e8d02d05ec1f87bb1cfc64d346ebccb14c71eb16ca072175de9"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 19 10:38:09.996361 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:09.996347    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7" podUID="46872ad2-8593-4f10-8d29-defbae3b3c50" containerName="service-proxy" containerID="cri-o://810be5ba67003e8d02d05ec1f87bb1cfc64d346ebccb14c71eb16ca072175de9" gracePeriod=30
Sep 19 10:38:10.521081 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:10.521048    2580 generic.go:356] "Generic (PLEG): container finished" podID="46872ad2-8593-4f10-8d29-defbae3b3c50" containerID="810be5ba67003e8d02d05ec1f87bb1cfc64d346ebccb14c71eb16ca072175de9" exitCode=2
Sep 19 10:38:10.521435 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:10.521120    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7" event={"ID":"46872ad2-8593-4f10-8d29-defbae3b3c50","Type":"ContainerDied","Data":"810be5ba67003e8d02d05ec1f87bb1cfc64d346ebccb14c71eb16ca072175de9"}
Sep 19 10:38:10.521435 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:10.521166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4959c68f-bhfs7" event={"ID":"46872ad2-8593-4f10-8d29-defbae3b3c50","Type":"ContainerStarted","Data":"41fddc60ee596931dafc7bda66fdedb2881514059524962e3f9870184d9c7348"}
Sep 19 10:38:10.666799 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:10.666738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:10.741304 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:10.741249    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:10.846189 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:10.846099    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:10.947690 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:10.947654    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:11.111699 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:11.111611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:11.212671 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:11.212635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:11.312680 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:11.312639    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:12.030907 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:12.030872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:14.926091 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:14.926048    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:16.333222 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:16.333185    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:16.391323 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:16.391288    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:16.919765 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:16.919707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:19.447970 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:19.447930    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:19.502263 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:19.502236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:26.598089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:26.598049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:26.699247 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:26.699212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:26.800245 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:26.800200    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:26.943046 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:26.943010    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:27.043930 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:27.043895    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:38:28.615015 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.614976    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69f679b797-cmc99" podUID="1181bcde-49d9-4582-9e19-c9603fc3c7cd" containerName="registry" containerID="cri-o://5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc" gracePeriod=30
Sep 19 10:38:28.631864 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:38:28.631836    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1181bcde_49d9_4582_9e19_c9603fc3c7cd.slice/crio-5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc.scope\": RecentStats: unable to find data in memory cache]"
Sep 19 10:38:28.860165 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.860133    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:38:28.941318 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.941283    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1181bcde-49d9-4582-9e19-c9603fc3c7cd-installation-pull-secrets\") pod \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") "
Sep 19 10:38:28.941318 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.941321    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls\") pod \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") "
Sep 19 10:38:28.941534 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.941346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-certificates\") pod \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") "
Sep 19 10:38:28.941534 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.941388    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1181bcde-49d9-4582-9e19-c9603fc3c7cd-image-registry-private-configuration\") pod \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") "
Sep 19 10:38:28.941534 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.941419    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-bound-sa-token\") pod \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") "
Sep 19 10:38:28.941534 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.941446    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1181bcde-49d9-4582-9e19-c9603fc3c7cd-trusted-ca\") pod \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") "
Sep 19 10:38:28.941534 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.941471    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1181bcde-49d9-4582-9e19-c9603fc3c7cd-ca-trust-extracted\") pod \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") "
Sep 19 10:38:28.941534 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.941506    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vszqz\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-kube-api-access-vszqz\") pod \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\" (UID: \"1181bcde-49d9-4582-9e19-c9603fc3c7cd\") "
Sep 19 10:38:28.942089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.942021    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1181bcde-49d9-4582-9e19-c9603fc3c7cd-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1181bcde-49d9-4582-9e19-c9603fc3c7cd" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:38:28.942089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.942040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1181bcde-49d9-4582-9e19-c9603fc3c7cd" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:38:28.944379 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.944343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1181bcde-49d9-4582-9e19-c9603fc3c7cd-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1181bcde-49d9-4582-9e19-c9603fc3c7cd" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:38:28.944379 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.944357    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-kube-api-access-vszqz" (OuterVolumeSpecName: "kube-api-access-vszqz") pod "1181bcde-49d9-4582-9e19-c9603fc3c7cd" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd"). InnerVolumeSpecName "kube-api-access-vszqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:38:28.944535 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.944418    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1181bcde-49d9-4582-9e19-c9603fc3c7cd-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1181bcde-49d9-4582-9e19-c9603fc3c7cd" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:38:28.944535 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.944475    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1181bcde-49d9-4582-9e19-c9603fc3c7cd" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:38:28.944608 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.944563    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1181bcde-49d9-4582-9e19-c9603fc3c7cd" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:38:28.950092 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:28.950066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1181bcde-49d9-4582-9e19-c9603fc3c7cd-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1181bcde-49d9-4582-9e19-c9603fc3c7cd" (UID: "1181bcde-49d9-4582-9e19-c9603fc3c7cd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:38:29.042880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.042838    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-certificates\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:38:29.042880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.042875    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1181bcde-49d9-4582-9e19-c9603fc3c7cd-image-registry-private-configuration\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:38:29.043094 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.042890    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-bound-sa-token\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:38:29.043094 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.042902    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1181bcde-49d9-4582-9e19-c9603fc3c7cd-trusted-ca\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:38:29.043094 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.042913    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1181bcde-49d9-4582-9e19-c9603fc3c7cd-ca-trust-extracted\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:38:29.043094 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.042925    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vszqz\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-kube-api-access-vszqz\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:38:29.043094 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.042935    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1181bcde-49d9-4582-9e19-c9603fc3c7cd-installation-pull-secrets\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:38:29.043094 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.042945    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181bcde-49d9-4582-9e19-c9603fc3c7cd-registry-tls\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:38:29.590772 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.590732    2580 generic.go:356] "Generic (PLEG): container finished" podID="1181bcde-49d9-4582-9e19-c9603fc3c7cd" containerID="5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc" exitCode=0
Sep 19 10:38:29.590966 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.590792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f679b797-cmc99" event={"ID":"1181bcde-49d9-4582-9e19-c9603fc3c7cd","Type":"ContainerDied","Data":"5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc"}
Sep 19 10:38:29.590966 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.590807    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f679b797-cmc99"
Sep 19 10:38:29.590966 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.590877    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f679b797-cmc99" event={"ID":"1181bcde-49d9-4582-9e19-c9603fc3c7cd","Type":"ContainerDied","Data":"daee7568632c64af5a4538453c7df322c853c1ef1ecbd21a401a8f5093e0e084"}
Sep 19 10:38:29.590966 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.590904    2580 scope.go:117] "RemoveContainer" containerID="5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc"
Sep 19 10:38:29.599553 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.599530    2580 scope.go:117] "RemoveContainer" containerID="5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc"
Sep 19 10:38:29.599862 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:38:29.599842    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc\": container with ID starting with 5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc not found: ID does not exist" containerID="5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc"
Sep 19 10:38:29.599915 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.599869    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc"} err="failed to get container status \"5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc\": rpc error: code = NotFound desc = could not find container \"5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc\": container with ID starting with 5bbccc0005b68afdc0125ca80ea95154a5ea64e9327a0a5211b22a3df3bc7adc not found: ID does not exist"
Sep 19 10:38:29.610448 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.610423    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69f679b797-cmc99"]
Sep 19 10:38:29.614445 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.614420    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69f679b797-cmc99"]
Sep 19 10:38:29.854185 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:38:29.854089    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1181bcde-49d9-4582-9e19-c9603fc3c7cd" path="/var/lib/kubelet/pods/1181bcde-49d9-4582-9e19-c9603fc3c7cd/volumes"
Sep 19 10:39:21.916586 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:21.916539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:39:21.967988 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:21.967942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:39:31.976015 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:31.975979    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:39:32.023889 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:32.023853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:39:32.123055 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:32.123026    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:39:32.172231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:32.172193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:39:32.932804 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:32.932772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:39:32.984334 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:32.984299    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:39:33.859602 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:33.859569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:39:33.914371 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:33.914340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:39:43.242283 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:43.242235    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:39:43.347275 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:43.347242    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:39:49.447078 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:49.447036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:39:49.547158 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:39:49.547117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:40:11.542545 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:40:11.542513    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:40:43.086608 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:40:43.086568    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:40:43.186662 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:40:43.186626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:40:55.755118 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:40:55.755086    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 10:40:55.760114 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:40:55.760082    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 19 10:40:55.765027 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:40:55.765006    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 10:41:04.635015 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:04.634977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:05.385791 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:05.385735    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:08.237287 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:08.237250    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:08.289231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:08.289204    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:09.762417 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:09.762374    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:09.871431 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:09.871384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:10.119984 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:10.119903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:10.216795 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:10.216768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:12.504898 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:12.504863    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:12.914228 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:12.914156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:14.835766 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:14.835724    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:14.886823 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:14.886786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:15.266953 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.266920    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx"]
Sep 19 10:41:15.267148 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.267136    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1181bcde-49d9-4582-9e19-c9603fc3c7cd" containerName="registry"
Sep 19 10:41:15.267190 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.267150    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1181bcde-49d9-4582-9e19-c9603fc3c7cd" containerName="registry"
Sep 19 10:41:15.267223 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.267198    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1181bcde-49d9-4582-9e19-c9603fc3c7cd" containerName="registry"
Sep 19 10:41:15.269828 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.269813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx"
Sep 19 10:41:15.272276 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.272253    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 19 10:41:15.272382 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.272262    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 19 10:41:15.272982 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.272959    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-j7786\""
Sep 19 10:41:15.278080 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.278057    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx"]
Sep 19 10:41:15.396640 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.396601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00014960-f4d3-4123-9a7b-9d4370ed84ea-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mtnsx\" (UID: \"00014960-f4d3-4123-9a7b-9d4370ed84ea\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx"
Sep 19 10:41:15.396832 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.396651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr4fv\" (UniqueName: \"kubernetes.io/projected/00014960-f4d3-4123-9a7b-9d4370ed84ea-kube-api-access-gr4fv\") pod \"cert-manager-cainjector-559cdd8c87-mtnsx\" (UID: \"00014960-f4d3-4123-9a7b-9d4370ed84ea\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx"
Sep 19 10:41:15.497467 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.497433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00014960-f4d3-4123-9a7b-9d4370ed84ea-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mtnsx\" (UID: \"00014960-f4d3-4123-9a7b-9d4370ed84ea\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx"
Sep 19 10:41:15.497637 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.497482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr4fv\" (UniqueName: \"kubernetes.io/projected/00014960-f4d3-4123-9a7b-9d4370ed84ea-kube-api-access-gr4fv\") pod \"cert-manager-cainjector-559cdd8c87-mtnsx\" (UID: \"00014960-f4d3-4123-9a7b-9d4370ed84ea\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx"
Sep 19 10:41:15.506602 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.506572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00014960-f4d3-4123-9a7b-9d4370ed84ea-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mtnsx\" (UID: \"00014960-f4d3-4123-9a7b-9d4370ed84ea\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx"
Sep 19 10:41:15.506975 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.506958    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr4fv\" (UniqueName: \"kubernetes.io/projected/00014960-f4d3-4123-9a7b-9d4370ed84ea-kube-api-access-gr4fv\") pod \"cert-manager-cainjector-559cdd8c87-mtnsx\" (UID: \"00014960-f4d3-4123-9a7b-9d4370ed84ea\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx"
Sep 19 10:41:15.579705 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.579624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx"
Sep 19 10:41:15.847731 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.847656    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx"]
Sep 19 10:41:15.851154 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:41:15.851124    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00014960_f4d3_4123_9a7b_9d4370ed84ea.slice/crio-a7e2400036a7db7ae726e767719cfa993b116c779ef14c15e1ef5dab10816883 WatchSource:0}: Error finding container a7e2400036a7db7ae726e767719cfa993b116c779ef14c15e1ef5dab10816883: Status 404 returned error can't find the container with id a7e2400036a7db7ae726e767719cfa993b116c779ef14c15e1ef5dab10816883
Sep 19 10:41:15.853045 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:15.853028    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 10:41:16.128665 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:16.128582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx" event={"ID":"00014960-f4d3-4123-9a7b-9d4370ed84ea","Type":"ContainerStarted","Data":"a7e2400036a7db7ae726e767719cfa993b116c779ef14c15e1ef5dab10816883"}
Sep 19 10:41:20.711426 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:20.711383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:21.027215 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:21.027121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:21.128343 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:21.128308    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:21.147416 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:21.147377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx" event={"ID":"00014960-f4d3-4123-9a7b-9d4370ed84ea","Type":"ContainerStarted","Data":"5f2de2012d0af85f138377ec566aff66d6a73462738cb2ccdaad36da2ad06559"}
Sep 19 10:41:21.162460 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:21.162412    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mtnsx" podStartSLOduration=1.756090385 podStartE2EDuration="6.162396263s" totalImagesPullingTime="4.406305878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:41:21.160931668 +0000 UTC m=+325.819676830" watchObservedRunningTime="2026-09-19 10:41:21.162396263 +0000 UTC m=+325.821141422"
Sep 19 10:41:26.454823 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:26.454783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:26.554255 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:26.554203    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:28.693519 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:28.693483    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll"]
Sep 19 10:41:28.696493 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:28.696477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll"
Sep 19 10:41:28.698639 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:28.698619    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 19 10:41:28.699375 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:28.699356    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 19 10:41:28.699451 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:28.699356    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zqjf2\""
Sep 19 10:41:28.704105 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:28.704081    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll"]
Sep 19 10:41:28.794559 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:28.794520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41988144-a03c-44a4-8729-d4b7d61116a8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-v5jll\" (UID: \"41988144-a03c-44a4-8729-d4b7d61116a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll"
Sep 19 10:41:28.794733 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:28.794584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk9jj\" (UniqueName: \"kubernetes.io/projected/41988144-a03c-44a4-8729-d4b7d61116a8-kube-api-access-vk9jj\") pod \"openshift-lws-operator-fd8ccff4c-v5jll\" (UID: \"41988144-a03c-44a4-8729-d4b7d61116a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll"
Sep 19 10:41:28.895692 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:28.895646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41988144-a03c-44a4-8729-d4b7d61116a8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-v5jll\" (UID: \"41988144-a03c-44a4-8729-d4b7d61116a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll"
Sep 19 10:41:28.895832 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:28.895714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk9jj\" (UniqueName: \"kubernetes.io/projected/41988144-a03c-44a4-8729-d4b7d61116a8-kube-api-access-vk9jj\") pod \"openshift-lws-operator-fd8ccff4c-v5jll\" (UID: \"41988144-a03c-44a4-8729-d4b7d61116a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll"
Sep 19 10:41:28.896068 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:28.896047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41988144-a03c-44a4-8729-d4b7d61116a8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-v5jll\" (UID: \"41988144-a03c-44a4-8729-d4b7d61116a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll"
Sep 19 10:41:28.904051 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:28.904026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk9jj\" (UniqueName: \"kubernetes.io/projected/41988144-a03c-44a4-8729-d4b7d61116a8-kube-api-access-vk9jj\") pod \"openshift-lws-operator-fd8ccff4c-v5jll\" (UID: \"41988144-a03c-44a4-8729-d4b7d61116a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll"
Sep 19 10:41:29.005011 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:29.004906    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll"
Sep 19 10:41:29.273577 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:29.273497    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll"]
Sep 19 10:41:29.276505 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:41:29.276477    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41988144_a03c_44a4_8729_d4b7d61116a8.slice/crio-df4ca826683e9d0ad83490b3c114734eefcf2c51fa3b46702f35b912e278e1f5 WatchSource:0}: Error finding container df4ca826683e9d0ad83490b3c114734eefcf2c51fa3b46702f35b912e278e1f5: Status 404 returned error can't find the container with id df4ca826683e9d0ad83490b3c114734eefcf2c51fa3b46702f35b912e278e1f5
Sep 19 10:41:30.178179 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:30.178143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll" event={"ID":"41988144-a03c-44a4-8729-d4b7d61116a8","Type":"ContainerStarted","Data":"df4ca826683e9d0ad83490b3c114734eefcf2c51fa3b46702f35b912e278e1f5"}
Sep 19 10:41:32.186680 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:32.186637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll" event={"ID":"41988144-a03c-44a4-8729-d4b7d61116a8","Type":"ContainerStarted","Data":"d494e93da4633ab7fc653af0956d12113d52f856521ffa135b56cf8aeaeff194"}
Sep 19 10:41:32.201721 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:32.201677    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v5jll" podStartSLOduration=1.913271341 podStartE2EDuration="4.201659816s" totalImagesPullingTime="2.288388475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:41:32.200956171 +0000 UTC m=+336.859701330" watchObservedRunningTime="2026-09-19 10:41:32.201659816 +0000 UTC m=+336.860404976"
Sep 19 10:41:32.865891 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:32.865852    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:33.890518 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:33.890481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:34.856633 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:34.856597    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:47.343658 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:47.343573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:47.444133 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:47.444095    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:47.958648 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:47.958609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:48.161744 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.161711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:48.263670 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.263582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:48.304972 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.304940    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"]
Sep 19 10:41:48.307923 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.307908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.310333 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.310295    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 19 10:41:48.310333 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.310319    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 19 10:41:48.310592 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.310346    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 19 10:41:48.310592 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.310394    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zrzqf\""
Sep 19 10:41:48.310689 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.310637    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 19 10:41:48.321800 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.321773    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"]
Sep 19 10:41:48.331578 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.331552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.331677 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.331585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-manifests\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.331677 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.331605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-tmp\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.331677 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.331628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.331804 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.331671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cn4c\" (UniqueName: \"kubernetes.io/projected/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-kube-api-access-5cn4c\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.418290 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.418252    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:48.432228 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.432202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-manifests\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.432325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.432238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-tmp\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.432325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.432262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.432325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.432301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cn4c\" (UniqueName: \"kubernetes.io/projected/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-kube-api-access-5cn4c\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.432433 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.432355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.432665 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.432644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-manifests\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.434797 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.434778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-tmp\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.434931 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.434913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.435090 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.435070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.449465 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.449434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cn4c\" (UniqueName: \"kubernetes.io/projected/43a85296-9b77-4c62-b5a5-ab4edc8e3e67-kube-api-access-5cn4c\") pod \"opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt\" (UID: \"43a85296-9b77-4c62-b5a5-ab4edc8e3e67\") " pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.512236 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.512202    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:48.612242 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.612149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:48.617858 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.617835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:48.681234 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.681205    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:48.745338 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.745296    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:48.846580 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.846547    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:48.895008 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:48.894984    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"]
Sep 19 10:41:48.897026 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:41:48.897002    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a85296_9b77_4c62_b5a5_ab4edc8e3e67.slice/crio-a91851552ad2df49c4ee5e953692c15407b7b82ef2bf7f727a171b3344eca72d WatchSource:0}: Error finding container a91851552ad2df49c4ee5e953692c15407b7b82ef2bf7f727a171b3344eca72d: Status 404 returned error can't find the container with id a91851552ad2df49c4ee5e953692c15407b7b82ef2bf7f727a171b3344eca72d
Sep 19 10:41:49.244319 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:49.244287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt" event={"ID":"43a85296-9b77-4c62-b5a5-ab4edc8e3e67","Type":"ContainerStarted","Data":"a91851552ad2df49c4ee5e953692c15407b7b82ef2bf7f727a171b3344eca72d"}
Sep 19 10:41:52.256650 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:52.256618    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt" event={"ID":"43a85296-9b77-4c62-b5a5-ab4edc8e3e67","Type":"ContainerStarted","Data":"a2285ea4eebdb8c404c7e2e0f437a8ea15d19e42be20ef320c77ba41ca24e1dc"}
Sep 19 10:41:53.261806 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:53.261704    2580 generic.go:356] "Generic (PLEG): container finished" podID="43a85296-9b77-4c62-b5a5-ab4edc8e3e67" containerID="a2285ea4eebdb8c404c7e2e0f437a8ea15d19e42be20ef320c77ba41ca24e1dc" exitCode=0
Sep 19 10:41:53.262166 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:53.261796    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt" event={"ID":"43a85296-9b77-4c62-b5a5-ab4edc8e3e67","Type":"ContainerDied","Data":"a2285ea4eebdb8c404c7e2e0f437a8ea15d19e42be20ef320c77ba41ca24e1dc"}
Sep 19 10:41:54.268170 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:54.268138    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt" event={"ID":"43a85296-9b77-4c62-b5a5-ab4edc8e3e67","Type":"ContainerStarted","Data":"0ec0cadb898c02a38cff0cc4369f658b004e24900dbed640f3f8ade4ffbd1ee8"}
Sep 19 10:41:54.268560 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:54.268264    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:41:54.290891 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:54.290840    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt" podStartSLOduration=1.801343973 podStartE2EDuration="6.290823967s" totalImagesPullingTime="3.432610174s" totalInitContainerRuntime="1.05686982s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:41:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:41:54.290673384 +0000 UTC m=+358.949418544" watchObservedRunningTime="2026-09-19 10:41:54.290823967 +0000 UTC m=+358.949569126"
Sep 19 10:41:55.487521 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:55.487474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:57.154187 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:57.154137    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:57.252300 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:57.252265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:58.790884 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:58.790842    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:41:58.888667 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:41:58.888631    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:05.273933 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:05.273897    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt"
Sep 19 10:42:06.672421 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:06.672384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:06.774722 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:06.774685    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:07.144413 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:07.144377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:07.249393 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:07.249359    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:17.780631 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:17.780597    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:17.834261 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:17.834223    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"]
Sep 19 10:42:17.837276 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:17.837258    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"
Sep 19 10:42:17.839258 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:17.839237    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 19 10:42:17.839347 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:17.839237    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 19 10:42:17.839394 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:17.839373    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 19 10:42:17.840168 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:17.840154    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rp7xj\""
Sep 19 10:42:17.840244 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:17.840161    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 19 10:42:17.847383 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:17.847362    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"]
Sep 19 10:42:17.850279 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:17.850232    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:17.953348 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:17.953317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa-tmp\") pod \"kube-auth-proxy-6d97bd96f8-tghnl\" (UID: \"688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa\") " pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"
Sep 19 10:42:17.953525 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:17.953356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69mhl\" (UniqueName: \"kubernetes.io/projected/688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa-kube-api-access-69mhl\") pod \"kube-auth-proxy-6d97bd96f8-tghnl\" (UID: \"688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa\") " pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"
Sep 19 10:42:17.953525 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:17.953379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa-tls-certs\") pod \"kube-auth-proxy-6d97bd96f8-tghnl\" (UID: \"688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa\") " pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"
Sep 19 10:42:18.054198 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.054109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa-tmp\") pod \"kube-auth-proxy-6d97bd96f8-tghnl\" (UID: \"688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa\") " pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"
Sep 19 10:42:18.054198 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.054151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mhl\" (UniqueName: \"kubernetes.io/projected/688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa-kube-api-access-69mhl\") pod \"kube-auth-proxy-6d97bd96f8-tghnl\" (UID: \"688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa\") " pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"
Sep 19 10:42:18.054198 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.054171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa-tls-certs\") pod \"kube-auth-proxy-6d97bd96f8-tghnl\" (UID: \"688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa\") " pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"
Sep 19 10:42:18.056516 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.056489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa-tmp\") pod \"kube-auth-proxy-6d97bd96f8-tghnl\" (UID: \"688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa\") " pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"
Sep 19 10:42:18.056825 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.056805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa-tls-certs\") pod \"kube-auth-proxy-6d97bd96f8-tghnl\" (UID: \"688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa\") " pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"
Sep 19 10:42:18.061889 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.061861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mhl\" (UniqueName: \"kubernetes.io/projected/688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa-kube-api-access-69mhl\") pod \"kube-auth-proxy-6d97bd96f8-tghnl\" (UID: \"688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa\") " pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"
Sep 19 10:42:18.146671 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.146635    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"
Sep 19 10:42:18.336141 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.336060    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h9wxr"]
Sep 19 10:42:18.339274 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.339256    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h9wxr"
Sep 19 10:42:18.341263 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.341236    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 19 10:42:18.341378 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.341342    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-cjsg2\""
Sep 19 10:42:18.346824 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.346798    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h9wxr"]
Sep 19 10:42:18.374165 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.374131    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:18.427487 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.427449    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl"]
Sep 19 10:42:18.432328 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:42:18.432294    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688d0c55_0ec7_4ed4_914e_53ba8f4ec0aa.slice/crio-1ad2f3d93cf255bf2853a1ea4f3dba74681aec0d3a122b58ee1584db0ec01a77 WatchSource:0}: Error finding container 1ad2f3d93cf255bf2853a1ea4f3dba74681aec0d3a122b58ee1584db0ec01a77: Status 404 returned error can't find the container with id 1ad2f3d93cf255bf2853a1ea4f3dba74681aec0d3a122b58ee1584db0ec01a77
Sep 19 10:42:18.443157 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.443122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:18.461854 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.460912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkkxf\" (UniqueName: \"kubernetes.io/projected/04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc-kube-api-access-nkkxf\") pod \"odh-model-controller-7998c67dc4-h9wxr\" (UID: \"04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc\") " pod="opendatahub/odh-model-controller-7998c67dc4-h9wxr"
Sep 19 10:42:18.461854 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.460977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc-cert\") pod \"odh-model-controller-7998c67dc4-h9wxr\" (UID: \"04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc\") " pod="opendatahub/odh-model-controller-7998c67dc4-h9wxr"
Sep 19 10:42:18.543377 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.543339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:18.562385 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.562342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkkxf\" (UniqueName: \"kubernetes.io/projected/04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc-kube-api-access-nkkxf\") pod \"odh-model-controller-7998c67dc4-h9wxr\" (UID: \"04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc\") " pod="opendatahub/odh-model-controller-7998c67dc4-h9wxr"
Sep 19 10:42:18.562558 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.562405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc-cert\") pod \"odh-model-controller-7998c67dc4-h9wxr\" (UID: \"04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc\") " pod="opendatahub/odh-model-controller-7998c67dc4-h9wxr"
Sep 19 10:42:18.565182 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.565153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc-cert\") pod \"odh-model-controller-7998c67dc4-h9wxr\" (UID: \"04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc\") " pod="opendatahub/odh-model-controller-7998c67dc4-h9wxr"
Sep 19 10:42:18.570566 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.570531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkkxf\" (UniqueName: \"kubernetes.io/projected/04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc-kube-api-access-nkkxf\") pod \"odh-model-controller-7998c67dc4-h9wxr\" (UID: \"04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc\") " pod="opendatahub/odh-model-controller-7998c67dc4-h9wxr"
Sep 19 10:42:18.646105 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.646011    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:18.649854 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.649835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h9wxr"
Sep 19 10:42:18.921280 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.921201    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"]
Sep 19 10:42:18.924119 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.924091    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:18.924559 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.924545    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:18.926825 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.926805    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 19 10:42:18.927007 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.926830    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 19 10:42:18.927116 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.927094    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 19 10:42:18.927221 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.927149    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 19 10:42:18.927313 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.927299    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2jqpc\""
Sep 19 10:42:18.927721 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.927702    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h9wxr"]
Sep 19 10:42:18.930802 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:42:18.930776    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c58eb8_f14d_4b7a_8fc5_433a2c13cfbc.slice/crio-29e5994a0a8b93fe1e54c4f071544c8f5e4e29ae6bc70cc7c01a7027ce85531e WatchSource:0}: Error finding container 29e5994a0a8b93fe1e54c4f071544c8f5e4e29ae6bc70cc7c01a7027ce85531e: Status 404 returned error can't find the container with id 29e5994a0a8b93fe1e54c4f071544c8f5e4e29ae6bc70cc7c01a7027ce85531e
Sep 19 10:42:18.935441 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.935418    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"]
Sep 19 10:42:18.986941 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:18.986902    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:19.067110 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.066689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01822f1-fb92-4cbd-8862-fdbca6a13803-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.067110 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.066739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v48nj\" (UniqueName: \"kubernetes.io/projected/b01822f1-fb92-4cbd-8862-fdbca6a13803-kube-api-access-v48nj\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.067110 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.066838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.067110 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.066895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.067110 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.066929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.067110 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.066981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.067110 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.067013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.150571 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.149183    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:19.168059 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.168018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01822f1-fb92-4cbd-8862-fdbca6a13803-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.168059 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.168063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v48nj\" (UniqueName: \"kubernetes.io/projected/b01822f1-fb92-4cbd-8862-fdbca6a13803-kube-api-access-v48nj\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.168294 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.168094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.168294 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.168127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.168294 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.168164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.168294 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.168224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.168294 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.168257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.168934 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.168907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.174070 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.173981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01822f1-fb92-4cbd-8862-fdbca6a13803-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.176001 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.175698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.176447 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.176399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.176680 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.176638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.184143 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.184093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.187387 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.187337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v48nj\" (UniqueName: \"kubernetes.io/projected/b01822f1-fb92-4cbd-8862-fdbca6a13803-kube-api-access-v48nj\") pod \"istiod-openshift-gateway-7b7995d5cd-jm2tc\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.234816 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.234779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:19.296408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.296180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:19.357939 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.357900    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h9wxr" event={"ID":"04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc","Type":"ContainerStarted","Data":"29e5994a0a8b93fe1e54c4f071544c8f5e4e29ae6bc70cc7c01a7027ce85531e"}
Sep 19 10:42:19.360779 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.360723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl" event={"ID":"688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa","Type":"ContainerStarted","Data":"1ad2f3d93cf255bf2853a1ea4f3dba74681aec0d3a122b58ee1584db0ec01a77"}
Sep 19 10:42:19.532122 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.532088    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"]
Sep 19 10:42:19.535237 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:42:19.535206    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01822f1_fb92_4cbd_8862_fdbca6a13803.slice/crio-c64b4a5790db0e9c92314ae9ad81c93d019f102fd865472e37a5286b5418cdca WatchSource:0}: Error finding container c64b4a5790db0e9c92314ae9ad81c93d019f102fd865472e37a5286b5418cdca: Status 404 returned error can't find the container with id c64b4a5790db0e9c92314ae9ad81c93d019f102fd865472e37a5286b5418cdca
Sep 19 10:42:19.866851 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:19.863981    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:20.367844 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:20.367745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc" event={"ID":"b01822f1-fb92-4cbd-8862-fdbca6a13803","Type":"ContainerStarted","Data":"c64b4a5790db0e9c92314ae9ad81c93d019f102fd865472e37a5286b5418cdca"}
Sep 19 10:42:23.386312 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:23.384997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h9wxr" event={"ID":"04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc","Type":"ContainerStarted","Data":"214d560a4be7f07441b59c125e57ee85fae2b10766f6db988a43fe336db8b4f6"}
Sep 19 10:42:23.386312 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:23.385844    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-h9wxr"
Sep 19 10:42:23.388631 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:23.388599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl" event={"ID":"688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa","Type":"ContainerStarted","Data":"e0d83487d86a9e41fbcadee4384a4a1323e2a2c88beac28bd9ebfe98b3a1fef6"}
Sep 19 10:42:23.401534 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:23.401415    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-h9wxr" podStartSLOduration=1.383781475 podStartE2EDuration="5.401395775s" totalImagesPullingTime="4.0176143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:42:23.400521167 +0000 UTC m=+388.059266338" watchObservedRunningTime="2026-09-19 10:42:23.401395775 +0000 UTC m=+388.060140935"
Sep 19 10:42:23.416565 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:23.416505    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6d97bd96f8-tghnl" podStartSLOduration=1.901993034 podStartE2EDuration="6.416482775s" totalImagesPullingTime="4.514489741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:42:23.415344157 +0000 UTC m=+388.074089321" watchObservedRunningTime="2026-09-19 10:42:23.416482775 +0000 UTC m=+388.075227937"
Sep 19 10:42:24.358238 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:24.358190    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:42:24.358325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:24.358281    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:42:24.891291 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:24.891247    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:24.991000 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:24.990964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:25.359394 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:25.359358    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:25.399312 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:25.399277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc" event={"ID":"b01822f1-fb92-4cbd-8862-fdbca6a13803","Type":"ContainerStarted","Data":"e420f3996d17773fbd8e258d227be598488238091678057b56d602413bf44d1a"}
Sep 19 10:42:25.399631 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:25.399613    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:25.401840 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:25.401812    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:25.419406 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:25.419358    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc" podStartSLOduration=2.59911181 podStartE2EDuration="7.419343784s" totalImagesPullingTime="4.820231974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:42:25.417308236 +0000 UTC m=+390.076053397" watchObservedRunningTime="2026-09-19 10:42:25.419343784 +0000 UTC m=+390.078088944"
Sep 19 10:42:25.568548 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:25.568510    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:25.775265 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:25.775231    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:29.123353 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.123319    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:29.172786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.172728    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"]
Sep 19 10:42:29.181969 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.181812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"
Sep 19 10:42:29.183194 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.183171    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw"]
Sep 19 10:42:29.184108 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.184090    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 19 10:42:29.184201 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.184099    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-4f28s\""
Sep 19 10:42:29.186526 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.186483    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw"
Sep 19 10:42:29.187093 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.187074    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"]
Sep 19 10:42:29.193227 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.193198    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2gjlx\""
Sep 19 10:42:29.193443 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.193316    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 19 10:42:29.202073 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.202053    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw"]
Sep 19 10:42:29.209429 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.209405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:29.359508 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.359463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/798963fa-d2b8-45c0-836e-4a8eb892c62e-cert\") pod \"kserve-controller-manager-d6f4b65b8-wwqqp\" (UID: \"798963fa-d2b8-45c0-836e-4a8eb892c62e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"
Sep 19 10:42:29.359508 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.359513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2ab3b71-ec29-4b92-8165-2fd479aba8ca-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6qkpw\" (UID: \"f2ab3b71-ec29-4b92-8165-2fd479aba8ca\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw"
Sep 19 10:42:29.359789 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.359585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvw4m\" (UniqueName: \"kubernetes.io/projected/798963fa-d2b8-45c0-836e-4a8eb892c62e-kube-api-access-nvw4m\") pod \"kserve-controller-manager-d6f4b65b8-wwqqp\" (UID: \"798963fa-d2b8-45c0-836e-4a8eb892c62e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"
Sep 19 10:42:29.359789 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.359642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zktzr\" (UniqueName: \"kubernetes.io/projected/f2ab3b71-ec29-4b92-8165-2fd479aba8ca-kube-api-access-zktzr\") pod \"llmisvc-controller-manager-777c6bd7d5-6qkpw\" (UID: \"f2ab3b71-ec29-4b92-8165-2fd479aba8ca\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw"
Sep 19 10:42:29.439464 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.439424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:29.461020 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.460987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/798963fa-d2b8-45c0-836e-4a8eb892c62e-cert\") pod \"kserve-controller-manager-d6f4b65b8-wwqqp\" (UID: \"798963fa-d2b8-45c0-836e-4a8eb892c62e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"
Sep 19 10:42:29.461020 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.461021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2ab3b71-ec29-4b92-8165-2fd479aba8ca-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6qkpw\" (UID: \"f2ab3b71-ec29-4b92-8165-2fd479aba8ca\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw"
Sep 19 10:42:29.461245 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.461045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvw4m\" (UniqueName: \"kubernetes.io/projected/798963fa-d2b8-45c0-836e-4a8eb892c62e-kube-api-access-nvw4m\") pod \"kserve-controller-manager-d6f4b65b8-wwqqp\" (UID: \"798963fa-d2b8-45c0-836e-4a8eb892c62e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"
Sep 19 10:42:29.461245 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.461121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zktzr\" (UniqueName: \"kubernetes.io/projected/f2ab3b71-ec29-4b92-8165-2fd479aba8ca-kube-api-access-zktzr\") pod \"llmisvc-controller-manager-777c6bd7d5-6qkpw\" (UID: \"f2ab3b71-ec29-4b92-8165-2fd479aba8ca\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw"
Sep 19 10:42:29.461245 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:42:29.461150    2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 19 10:42:29.461245 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:42:29.461218    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/798963fa-d2b8-45c0-836e-4a8eb892c62e-cert podName:798963fa-d2b8-45c0-836e-4a8eb892c62e nodeName:}" failed. No retries permitted until 2026-09-19 10:42:29.961196861 +0000 UTC m=+394.619942005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/798963fa-d2b8-45c0-836e-4a8eb892c62e-cert") pod "kserve-controller-manager-d6f4b65b8-wwqqp" (UID: "798963fa-d2b8-45c0-836e-4a8eb892c62e") : secret "kserve-webhook-server-cert" not found
Sep 19 10:42:29.463786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.463746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2ab3b71-ec29-4b92-8165-2fd479aba8ca-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6qkpw\" (UID: \"f2ab3b71-ec29-4b92-8165-2fd479aba8ca\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw"
Sep 19 10:42:29.479154 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.479124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zktzr\" (UniqueName: \"kubernetes.io/projected/f2ab3b71-ec29-4b92-8165-2fd479aba8ca-kube-api-access-zktzr\") pod \"llmisvc-controller-manager-777c6bd7d5-6qkpw\" (UID: \"f2ab3b71-ec29-4b92-8165-2fd479aba8ca\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw"
Sep 19 10:42:29.480385 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.480364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvw4m\" (UniqueName: \"kubernetes.io/projected/798963fa-d2b8-45c0-836e-4a8eb892c62e-kube-api-access-nvw4m\") pod \"kserve-controller-manager-d6f4b65b8-wwqqp\" (UID: \"798963fa-d2b8-45c0-836e-4a8eb892c62e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"
Sep 19 10:42:29.500466 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.500432    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw"
Sep 19 10:42:29.545992 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.545957    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:29.658636 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.658606    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:29.773129 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.773029    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw"]
Sep 19 10:42:29.775589 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:42:29.775566    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf2ab3b71_ec29_4b92_8165_2fd479aba8ca.slice/crio-059544377ce81eda7bc872c3e538ed9a5742865524b88b99b2b00d6ecf2baed0 WatchSource:0}: Error finding container 059544377ce81eda7bc872c3e538ed9a5742865524b88b99b2b00d6ecf2baed0: Status 404 returned error can't find the container with id 059544377ce81eda7bc872c3e538ed9a5742865524b88b99b2b00d6ecf2baed0
Sep 19 10:42:29.966161 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.966123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/798963fa-d2b8-45c0-836e-4a8eb892c62e-cert\") pod \"kserve-controller-manager-d6f4b65b8-wwqqp\" (UID: \"798963fa-d2b8-45c0-836e-4a8eb892c62e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"
Sep 19 10:42:29.968640 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:29.968612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/798963fa-d2b8-45c0-836e-4a8eb892c62e-cert\") pod \"kserve-controller-manager-d6f4b65b8-wwqqp\" (UID: \"798963fa-d2b8-45c0-836e-4a8eb892c62e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"
Sep 19 10:42:30.093934 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:30.093833    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"
Sep 19 10:42:30.397162 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:30.397138    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"]
Sep 19 10:42:30.422226 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:30.422180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw" event={"ID":"f2ab3b71-ec29-4b92-8165-2fd479aba8ca","Type":"ContainerStarted","Data":"059544377ce81eda7bc872c3e538ed9a5742865524b88b99b2b00d6ecf2baed0"}
Sep 19 10:42:30.438112 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:42:30.438081    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798963fa_d2b8_45c0_836e_4a8eb892c62e.slice/crio-4ff2c30c49e8e76e4fc99528fa01a2ef694f7a85b61ca81c6d0dfd40f8b63956 WatchSource:0}: Error finding container 4ff2c30c49e8e76e4fc99528fa01a2ef694f7a85b61ca81c6d0dfd40f8b63956: Status 404 returned error can't find the container with id 4ff2c30c49e8e76e4fc99528fa01a2ef694f7a85b61ca81c6d0dfd40f8b63956
Sep 19 10:42:31.429151 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:31.429112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp" event={"ID":"798963fa-d2b8-45c0-836e-4a8eb892c62e","Type":"ContainerStarted","Data":"4ff2c30c49e8e76e4fc99528fa01a2ef694f7a85b61ca81c6d0dfd40f8b63956"}
Sep 19 10:42:33.020313 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.020278    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"]
Sep 19 10:42:33.020800 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.020507    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc" podUID="b01822f1-fb92-4cbd-8862-fdbca6a13803" containerName="discovery" containerID="cri-o://e420f3996d17773fbd8e258d227be598488238091678057b56d602413bf44d1a" gracePeriod=30
Sep 19 10:42:33.260599 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.260572    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:33.396902 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.396810    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01822f1-fb92-4cbd-8862-fdbca6a13803-local-certs\") pod \"b01822f1-fb92-4cbd-8862-fdbca6a13803\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") "
Sep 19 10:42:33.396902 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.396865    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-csr-ca-configmap\") pod \"b01822f1-fb92-4cbd-8862-fdbca6a13803\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") "
Sep 19 10:42:33.396902 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.396888    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-kubeconfig\") pod \"b01822f1-fb92-4cbd-8862-fdbca6a13803\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") "
Sep 19 10:42:33.397184 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.396916    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-token\") pod \"b01822f1-fb92-4cbd-8862-fdbca6a13803\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") "
Sep 19 10:42:33.397184 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.396936    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-csr-dns-cert\") pod \"b01822f1-fb92-4cbd-8862-fdbca6a13803\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") "
Sep 19 10:42:33.397184 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.396965    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v48nj\" (UniqueName: \"kubernetes.io/projected/b01822f1-fb92-4cbd-8862-fdbca6a13803-kube-api-access-v48nj\") pod \"b01822f1-fb92-4cbd-8862-fdbca6a13803\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") "
Sep 19 10:42:33.397184 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.397003    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-cacerts\") pod \"b01822f1-fb92-4cbd-8862-fdbca6a13803\" (UID: \"b01822f1-fb92-4cbd-8862-fdbca6a13803\") "
Sep 19 10:42:33.397383 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.397325    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b01822f1-fb92-4cbd-8862-fdbca6a13803" (UID: "b01822f1-fb92-4cbd-8862-fdbca6a13803"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:42:33.399383 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.399354    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01822f1-fb92-4cbd-8862-fdbca6a13803-local-certs" (OuterVolumeSpecName: "local-certs") pod "b01822f1-fb92-4cbd-8862-fdbca6a13803" (UID: "b01822f1-fb92-4cbd-8862-fdbca6a13803"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:42:33.399383 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.399361    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b01822f1-fb92-4cbd-8862-fdbca6a13803" (UID: "b01822f1-fb92-4cbd-8862-fdbca6a13803"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:42:33.399599 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.399580    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b01822f1-fb92-4cbd-8862-fdbca6a13803" (UID: "b01822f1-fb92-4cbd-8862-fdbca6a13803"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:42:33.399599 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.399584    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-token" (OuterVolumeSpecName: "istio-token") pod "b01822f1-fb92-4cbd-8862-fdbca6a13803" (UID: "b01822f1-fb92-4cbd-8862-fdbca6a13803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:42:33.399890 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.399868    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-cacerts" (OuterVolumeSpecName: "cacerts") pod "b01822f1-fb92-4cbd-8862-fdbca6a13803" (UID: "b01822f1-fb92-4cbd-8862-fdbca6a13803"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:42:33.399890 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.399878    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01822f1-fb92-4cbd-8862-fdbca6a13803-kube-api-access-v48nj" (OuterVolumeSpecName: "kube-api-access-v48nj") pod "b01822f1-fb92-4cbd-8862-fdbca6a13803" (UID: "b01822f1-fb92-4cbd-8862-fdbca6a13803"). InnerVolumeSpecName "kube-api-access-v48nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:42:33.443838 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.443803    2580 generic.go:356] "Generic (PLEG): container finished" podID="b01822f1-fb92-4cbd-8862-fdbca6a13803" containerID="e420f3996d17773fbd8e258d227be598488238091678057b56d602413bf44d1a" exitCode=0
Sep 19 10:42:33.443978 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.443848    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"
Sep 19 10:42:33.443978 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.443875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc" event={"ID":"b01822f1-fb92-4cbd-8862-fdbca6a13803","Type":"ContainerDied","Data":"e420f3996d17773fbd8e258d227be598488238091678057b56d602413bf44d1a"}
Sep 19 10:42:33.443978 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.443922    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc" event={"ID":"b01822f1-fb92-4cbd-8862-fdbca6a13803","Type":"ContainerDied","Data":"c64b4a5790db0e9c92314ae9ad81c93d019f102fd865472e37a5286b5418cdca"}
Sep 19 10:42:33.443978 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.443943    2580 scope.go:117] "RemoveContainer" containerID="e420f3996d17773fbd8e258d227be598488238091678057b56d602413bf44d1a"
Sep 19 10:42:33.445517 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.445481    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp" event={"ID":"798963fa-d2b8-45c0-836e-4a8eb892c62e","Type":"ContainerStarted","Data":"ce2ed77babcb04f73fc6d166c954a5882fe6cb31dd7da72bac4489cf28b5187d"}
Sep 19 10:42:33.445623 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.445543    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"
Sep 19 10:42:33.447163 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.447141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw" event={"ID":"f2ab3b71-ec29-4b92-8165-2fd479aba8ca","Type":"ContainerStarted","Data":"2d228f29820bf97423b4f1e51a14671b2632f9ea692e5298415ba82549a5b271"}
Sep 19 10:42:33.447263 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.447221    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw"
Sep 19 10:42:33.452767 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.452736    2580 scope.go:117] "RemoveContainer" containerID="e420f3996d17773fbd8e258d227be598488238091678057b56d602413bf44d1a"
Sep 19 10:42:33.453044 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:42:33.453027    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e420f3996d17773fbd8e258d227be598488238091678057b56d602413bf44d1a\": container with ID starting with e420f3996d17773fbd8e258d227be598488238091678057b56d602413bf44d1a not found: ID does not exist" containerID="e420f3996d17773fbd8e258d227be598488238091678057b56d602413bf44d1a"
Sep 19 10:42:33.453092 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.453052    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e420f3996d17773fbd8e258d227be598488238091678057b56d602413bf44d1a"} err="failed to get container status \"e420f3996d17773fbd8e258d227be598488238091678057b56d602413bf44d1a\": rpc error: code = NotFound desc = could not find container \"e420f3996d17773fbd8e258d227be598488238091678057b56d602413bf44d1a\": container with ID starting with e420f3996d17773fbd8e258d227be598488238091678057b56d602413bf44d1a not found: ID does not exist"
Sep 19 10:42:33.463583 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.463536    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw" podStartSLOduration=1.394561002 podStartE2EDuration="4.463517431s" totalImagesPullingTime="3.068956429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:42:33.462246377 +0000 UTC m=+398.120991533" watchObservedRunningTime="2026-09-19 10:42:33.463517431 +0000 UTC m=+398.122262592"
Sep 19 10:42:33.478431 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.478361    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp" podStartSLOduration=2.070484718 podStartE2EDuration="4.478344701s" totalImagesPullingTime="2.407859983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:42:33.477573898 +0000 UTC m=+398.136319058" watchObservedRunningTime="2026-09-19 10:42:33.478344701 +0000 UTC m=+398.137089861"
Sep 19 10:42:33.492095 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.492065    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"]
Sep 19 10:42:33.495374 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.495354    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jm2tc"]
Sep 19 10:42:33.498354 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.498331    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-csr-ca-configmap\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:42:33.498410 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.498359    2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-kubeconfig\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:42:33.498410 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.498370    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-token\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:42:33.498410 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.498378    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-istio-csr-dns-cert\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:42:33.498410 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.498386    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v48nj\" (UniqueName: \"kubernetes.io/projected/b01822f1-fb92-4cbd-8862-fdbca6a13803-kube-api-access-v48nj\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:42:33.498410 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.498396    2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b01822f1-fb92-4cbd-8862-fdbca6a13803-cacerts\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:42:33.498410 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.498407    2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01822f1-fb92-4cbd-8862-fdbca6a13803-local-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:42:33.857742 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:33.857701    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b01822f1-fb92-4cbd-8862-fdbca6a13803" path="/var/lib/kubelet/pods/b01822f1-fb92-4cbd-8862-fdbca6a13803/volumes"
Sep 19 10:42:35.401415 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:35.401384    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-h9wxr"
Sep 19 10:42:46.834811 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:46.834742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:46.882662 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:46.882612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:46.972832 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:46.972799    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:47.026632 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:47.026592    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:47.573947 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:47.573908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:47.621024 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:47.620976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:51.357294 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:51.357257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:51.413668 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:51.413630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:53.312710 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:53.312667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:53.409449 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:53.409417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:58.192804 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:58.192739    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:58.242636 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:58.242597    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:42:58.297780 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:42:58.297721    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:04.453486 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:04.453459    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6qkpw"
Sep 19 10:43:04.456535 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:04.456515    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wwqqp"
Sep 19 10:43:19.298221 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:19.298126    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:19.397291 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:19.397258    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:31.669232 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:31.669187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:32.463041 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:32.463006    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:34.939078 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:34.939045    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-jnvrj"]
Sep 19 10:43:34.939483 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:34.939314    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01822f1-fb92-4cbd-8862-fdbca6a13803" containerName="discovery"
Sep 19 10:43:34.939483 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:34.939324    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01822f1-fb92-4cbd-8862-fdbca6a13803" containerName="discovery"
Sep 19 10:43:34.939483 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:34.939371    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01822f1-fb92-4cbd-8862-fdbca6a13803" containerName="discovery"
Sep 19 10:43:34.942225 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:34.942203    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-jnvrj"
Sep 19 10:43:34.946603 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:34.946581    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xk982\""
Sep 19 10:43:34.946723 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:34.946640    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 19 10:43:34.946723 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:34.946653    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 19 10:43:34.953432 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:34.953408    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-jnvrj"]
Sep 19 10:43:35.041075 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:35.041033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qlsq\" (UniqueName: \"kubernetes.io/projected/c3539e5e-710c-4b48-8205-92dd4b67f8ca-kube-api-access-6qlsq\") pod \"authorino-operator-76b478bd78-jnvrj\" (UID: \"c3539e5e-710c-4b48-8205-92dd4b67f8ca\") " pod="kuadrant-system/authorino-operator-76b478bd78-jnvrj"
Sep 19 10:43:35.141613 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:35.141578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qlsq\" (UniqueName: \"kubernetes.io/projected/c3539e5e-710c-4b48-8205-92dd4b67f8ca-kube-api-access-6qlsq\") pod \"authorino-operator-76b478bd78-jnvrj\" (UID: \"c3539e5e-710c-4b48-8205-92dd4b67f8ca\") " pod="kuadrant-system/authorino-operator-76b478bd78-jnvrj"
Sep 19 10:43:35.153930 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:35.153904    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qlsq\" (UniqueName: \"kubernetes.io/projected/c3539e5e-710c-4b48-8205-92dd4b67f8ca-kube-api-access-6qlsq\") pod \"authorino-operator-76b478bd78-jnvrj\" (UID: \"c3539e5e-710c-4b48-8205-92dd4b67f8ca\") " pod="kuadrant-system/authorino-operator-76b478bd78-jnvrj"
Sep 19 10:43:35.252405 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:35.252317    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-jnvrj"
Sep 19 10:43:35.524885 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:35.524810    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-jnvrj"]
Sep 19 10:43:35.527749 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:43:35.527723    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3539e5e_710c_4b48_8205_92dd4b67f8ca.slice/crio-2754a626762e95bbbb1881e840dba93e457d8409dd7f531bf9b1d56fee56afce WatchSource:0}: Error finding container 2754a626762e95bbbb1881e840dba93e457d8409dd7f531bf9b1d56fee56afce: Status 404 returned error can't find the container with id 2754a626762e95bbbb1881e840dba93e457d8409dd7f531bf9b1d56fee56afce
Sep 19 10:43:35.670868 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:35.670829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:35.710095 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:35.710064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-jnvrj" event={"ID":"c3539e5e-710c-4b48-8205-92dd4b67f8ca","Type":"ContainerStarted","Data":"2754a626762e95bbbb1881e840dba93e457d8409dd7f531bf9b1d56fee56afce"}
Sep 19 10:43:35.922011 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:35.921908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:36.026396 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:36.026357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:37.136585 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:37.136552    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"]
Sep 19 10:43:37.144970 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:37.144941    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"
Sep 19 10:43:37.147296 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:37.147273    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-sgwbv\""
Sep 19 10:43:37.148213 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:37.148189    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"]
Sep 19 10:43:37.257222 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:37.257181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgz94\" (UniqueName: \"kubernetes.io/projected/01b44ae0-9e72-4493-b89a-8c7a1d5db9da-kube-api-access-hgz94\") pod \"limitador-operator-controller-manager-74495c69f8-wxr7n\" (UID: \"01b44ae0-9e72-4493-b89a-8c7a1d5db9da\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"
Sep 19 10:43:37.358509 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:37.358472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgz94\" (UniqueName: \"kubernetes.io/projected/01b44ae0-9e72-4493-b89a-8c7a1d5db9da-kube-api-access-hgz94\") pod \"limitador-operator-controller-manager-74495c69f8-wxr7n\" (UID: \"01b44ae0-9e72-4493-b89a-8c7a1d5db9da\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"
Sep 19 10:43:37.376669 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:37.376641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgz94\" (UniqueName: \"kubernetes.io/projected/01b44ae0-9e72-4493-b89a-8c7a1d5db9da-kube-api-access-hgz94\") pod \"limitador-operator-controller-manager-74495c69f8-wxr7n\" (UID: \"01b44ae0-9e72-4493-b89a-8c7a1d5db9da\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"
Sep 19 10:43:37.458314 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:37.458280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"
Sep 19 10:43:37.893275 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:37.893235    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:37.950426 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:37.950392    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"]
Sep 19 10:43:37.953521 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:43:37.953492    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b44ae0_9e72_4493_b89a_8c7a1d5db9da.slice/crio-2043986ab8bd479f086f1223950e706764eb55ae19f55275800bc7a885567220 WatchSource:0}: Error finding container 2043986ab8bd479f086f1223950e706764eb55ae19f55275800bc7a885567220: Status 404 returned error can't find the container with id 2043986ab8bd479f086f1223950e706764eb55ae19f55275800bc7a885567220
Sep 19 10:43:38.114084 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:38.113999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:38.725289 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:38.725248    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n" event={"ID":"01b44ae0-9e72-4493-b89a-8c7a1d5db9da","Type":"ContainerStarted","Data":"2043986ab8bd479f086f1223950e706764eb55ae19f55275800bc7a885567220"}
Sep 19 10:43:38.727280 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:38.727244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-jnvrj" event={"ID":"c3539e5e-710c-4b48-8205-92dd4b67f8ca","Type":"ContainerStarted","Data":"1d1a653281829421d32c363bab7cbcad88a4ee68703738b96f2aad96bc48f20c"}
Sep 19 10:43:38.727416 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:38.727341    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-jnvrj"
Sep 19 10:43:38.747523 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:38.747462    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-jnvrj" podStartSLOduration=2.5326852410000003 podStartE2EDuration="4.747441549s" totalImagesPullingTime="2.214756308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:43:38.745376873 +0000 UTC m=+463.404122035" watchObservedRunningTime="2026-09-19 10:43:38.747441549 +0000 UTC m=+463.406186711"
Sep 19 10:43:39.462222 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:39.462142    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6ptv6"]
Sep 19 10:43:39.465629 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:39.465607    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6ptv6"
Sep 19 10:43:39.468245 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:39.468226    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mr55h\""
Sep 19 10:43:39.468545 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:39.468529    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 19 10:43:39.476048 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:39.476027    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6ptv6"]
Sep 19 10:43:39.577035 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:39.577000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2fh6\" (UniqueName: \"kubernetes.io/projected/338f0495-8618-4e40-a96b-b74298ec67d1-kube-api-access-m2fh6\") pod \"dns-operator-controller-manager-7b84cd7c9c-6ptv6\" (UID: \"338f0495-8618-4e40-a96b-b74298ec67d1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6ptv6"
Sep 19 10:43:39.678050 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:39.678012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2fh6\" (UniqueName: \"kubernetes.io/projected/338f0495-8618-4e40-a96b-b74298ec67d1-kube-api-access-m2fh6\") pod \"dns-operator-controller-manager-7b84cd7c9c-6ptv6\" (UID: \"338f0495-8618-4e40-a96b-b74298ec67d1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6ptv6"
Sep 19 10:43:39.688894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:39.688862    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2fh6\" (UniqueName: \"kubernetes.io/projected/338f0495-8618-4e40-a96b-b74298ec67d1-kube-api-access-m2fh6\") pod \"dns-operator-controller-manager-7b84cd7c9c-6ptv6\" (UID: \"338f0495-8618-4e40-a96b-b74298ec67d1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6ptv6"
Sep 19 10:43:39.732910 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:39.732820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n" event={"ID":"01b44ae0-9e72-4493-b89a-8c7a1d5db9da","Type":"ContainerStarted","Data":"2716441a0846f932d0bd63691b590bb4de6e61fdef1f48071fb729f17b3ddff4"}
Sep 19 10:43:39.733269 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:39.733076    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"
Sep 19 10:43:39.750993 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:39.750947    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n" podStartSLOduration=1.507902903 podStartE2EDuration="2.750931158s" totalImagesPullingTime="1.243028255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:43:39.749649505 +0000 UTC m=+464.408394664" watchObservedRunningTime="2026-09-19 10:43:39.750931158 +0000 UTC m=+464.409676318"
Sep 19 10:43:39.776260 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:39.776225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6ptv6"
Sep 19 10:43:39.884786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:39.884733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:40.049267 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:40.049243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6ptv6"]
Sep 19 10:43:40.051654 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:43:40.051628    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338f0495_8618_4e40_a96b_b74298ec67d1.slice/crio-64199c48370bca410be35179cdd62288008e392f5024db643fcf99cd56d47663 WatchSource:0}: Error finding container 64199c48370bca410be35179cdd62288008e392f5024db643fcf99cd56d47663: Status 404 returned error can't find the container with id 64199c48370bca410be35179cdd62288008e392f5024db643fcf99cd56d47663
Sep 19 10:43:40.266167 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:40.266136    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:40.520429 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:40.520391    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:40.662785 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:40.662716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:40.739765 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:40.739708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6ptv6" event={"ID":"338f0495-8618-4e40-a96b-b74298ec67d1","Type":"ContainerStarted","Data":"64199c48370bca410be35179cdd62288008e392f5024db643fcf99cd56d47663"}
Sep 19 10:43:41.720141 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:41.720110    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:41.745310 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:41.745222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6ptv6" event={"ID":"338f0495-8618-4e40-a96b-b74298ec67d1","Type":"ContainerStarted","Data":"3a0d03dc7d0bbb6d1fbb3de92f1e58da25b9a9f58e7db6b08139b1e17219933b"}
Sep 19 10:43:41.745310 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:41.745300    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6ptv6"
Sep 19 10:43:41.764170 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:41.764103    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6ptv6" podStartSLOduration=1.341874027 podStartE2EDuration="2.764086967s" totalImagesPullingTime="1.42221294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:43:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:43:41.762825595 +0000 UTC m=+466.421570758" watchObservedRunningTime="2026-09-19 10:43:41.764086967 +0000 UTC m=+466.422832126"
Sep 19 10:43:43.646341 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:43.646305    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"]
Sep 19 10:43:43.649533 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:43.649514    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:43:43.651825 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:43.651802    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cxgff\""
Sep 19 10:43:43.661007 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:43.660984    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"]
Sep 19 10:43:43.708949 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:43.708903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvsln\" (UniqueName: \"kubernetes.io/projected/0180753f-d097-49dc-80ed-e8c540e9d7cd-kube-api-access-hvsln\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ccz6l\" (UID: \"0180753f-d097-49dc-80ed-e8c540e9d7cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:43:43.709137 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:43.708961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0180753f-d097-49dc-80ed-e8c540e9d7cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ccz6l\" (UID: \"0180753f-d097-49dc-80ed-e8c540e9d7cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:43:43.809969 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:43.809931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvsln\" (UniqueName: \"kubernetes.io/projected/0180753f-d097-49dc-80ed-e8c540e9d7cd-kube-api-access-hvsln\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ccz6l\" (UID: \"0180753f-d097-49dc-80ed-e8c540e9d7cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:43:43.810161 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:43.809979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0180753f-d097-49dc-80ed-e8c540e9d7cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ccz6l\" (UID: \"0180753f-d097-49dc-80ed-e8c540e9d7cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:43:43.810308 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:43.810288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0180753f-d097-49dc-80ed-e8c540e9d7cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ccz6l\" (UID: \"0180753f-d097-49dc-80ed-e8c540e9d7cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:43:43.818746 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:43.818724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvsln\" (UniqueName: \"kubernetes.io/projected/0180753f-d097-49dc-80ed-e8c540e9d7cd-kube-api-access-hvsln\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ccz6l\" (UID: \"0180753f-d097-49dc-80ed-e8c540e9d7cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:43:43.959507 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:43.959476    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:43:44.241478 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:44.241450    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"]
Sep 19 10:43:44.244354 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:43:44.244323    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0180753f_d097_49dc_80ed_e8c540e9d7cd.slice/crio-8a6e02484f24bb98f2af29b51a043e8f12cd5d313faaf18ad22b27b696d1c29c WatchSource:0}: Error finding container 8a6e02484f24bb98f2af29b51a043e8f12cd5d313faaf18ad22b27b696d1c29c: Status 404 returned error can't find the container with id 8a6e02484f24bb98f2af29b51a043e8f12cd5d313faaf18ad22b27b696d1c29c
Sep 19 10:43:44.760021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:44.759982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l" event={"ID":"0180753f-d097-49dc-80ed-e8c540e9d7cd","Type":"ContainerStarted","Data":"8a6e02484f24bb98f2af29b51a043e8f12cd5d313faaf18ad22b27b696d1c29c"}
Sep 19 10:43:45.679776 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:45.679710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:45.778340 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:45.778301    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:46.269149 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:46.269116    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:46.369413 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:46.369375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:48.779933 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:48.779895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l" event={"ID":"0180753f-d097-49dc-80ed-e8c540e9d7cd","Type":"ContainerStarted","Data":"939b88c83b92da9e9c2a934d478fc8f0ba240d6d3666a8ffdb57d3a392602b2a"}
Sep 19 10:43:48.780318 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:48.780005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:43:48.799535 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:48.799473    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l" podStartSLOduration=1.956987867 podStartE2EDuration="5.799457964s" totalImagesPullingTime="3.842470097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:43:48.798043136 +0000 UTC m=+473.456788297" watchObservedRunningTime="2026-09-19 10:43:48.799457964 +0000 UTC m=+473.458203124"
Sep 19 10:43:49.675034 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:49.674999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:49.735878 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:49.735845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-jnvrj"
Sep 19 10:43:49.759538 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:49.759468    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:50.742457 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.742425    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"
Sep 19 10:43:50.750541 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.750510    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"]
Sep 19 10:43:50.751366 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.751340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:50.753981 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.753965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"
Sep 19 10:43:50.755930 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.755910    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 19 10:43:50.756044 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.755962    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 19 10:43:50.756044 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.755971    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ss7b2\""
Sep 19 10:43:50.764858 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.764832    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"]
Sep 19 10:43:50.871051 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.871009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4fcae889-b4f9-4446-92a5-cc8715472411-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-wzqdr\" (UID: \"4fcae889-b4f9-4446-92a5-cc8715472411\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"
Sep 19 10:43:50.871247 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.871073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fcae889-b4f9-4446-92a5-cc8715472411-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wzqdr\" (UID: \"4fcae889-b4f9-4446-92a5-cc8715472411\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"
Sep 19 10:43:50.871247 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.871144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rvql\" (UniqueName: \"kubernetes.io/projected/4fcae889-b4f9-4446-92a5-cc8715472411-kube-api-access-8rvql\") pod \"kuadrant-console-plugin-fbdd4f445-wzqdr\" (UID: \"4fcae889-b4f9-4446-92a5-cc8715472411\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"
Sep 19 10:43:50.946473 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.946438    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:50.972443 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.972410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4fcae889-b4f9-4446-92a5-cc8715472411-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-wzqdr\" (UID: \"4fcae889-b4f9-4446-92a5-cc8715472411\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"
Sep 19 10:43:50.972592 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.972457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fcae889-b4f9-4446-92a5-cc8715472411-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wzqdr\" (UID: \"4fcae889-b4f9-4446-92a5-cc8715472411\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"
Sep 19 10:43:50.972592 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.972512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvql\" (UniqueName: \"kubernetes.io/projected/4fcae889-b4f9-4446-92a5-cc8715472411-kube-api-access-8rvql\") pod \"kuadrant-console-plugin-fbdd4f445-wzqdr\" (UID: \"4fcae889-b4f9-4446-92a5-cc8715472411\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"
Sep 19 10:43:50.973222 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.973203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4fcae889-b4f9-4446-92a5-cc8715472411-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-wzqdr\" (UID: \"4fcae889-b4f9-4446-92a5-cc8715472411\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"
Sep 19 10:43:50.975101 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.975076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fcae889-b4f9-4446-92a5-cc8715472411-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wzqdr\" (UID: \"4fcae889-b4f9-4446-92a5-cc8715472411\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"
Sep 19 10:43:50.981253 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:50.981230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvql\" (UniqueName: \"kubernetes.io/projected/4fcae889-b4f9-4446-92a5-cc8715472411-kube-api-access-8rvql\") pod \"kuadrant-console-plugin-fbdd4f445-wzqdr\" (UID: \"4fcae889-b4f9-4446-92a5-cc8715472411\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"
Sep 19 10:43:51.063193 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:51.063099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"
Sep 19 10:43:51.282540 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:51.282502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:51.339723 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:51.339636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr"]
Sep 19 10:43:51.342773 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:43:51.342713    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fcae889_b4f9_4446_92a5_cc8715472411.slice/crio-8e67519b1544006e3c3ef279b5ddbc99f391ecf1814ac5ce8a1ff0a44442c6ca WatchSource:0}: Error finding container 8e67519b1544006e3c3ef279b5ddbc99f391ecf1814ac5ce8a1ff0a44442c6ca: Status 404 returned error can't find the container with id 8e67519b1544006e3c3ef279b5ddbc99f391ecf1814ac5ce8a1ff0a44442c6ca
Sep 19 10:43:51.342922 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:51.342891    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:51.794523 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:51.794490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr" event={"ID":"4fcae889-b4f9-4446-92a5-cc8715472411","Type":"ContainerStarted","Data":"8e67519b1544006e3c3ef279b5ddbc99f391ecf1814ac5ce8a1ff0a44442c6ca"}
Sep 19 10:43:52.753146 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:52.753112    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6ptv6"
Sep 19 10:43:56.821823 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:56.821785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr" event={"ID":"4fcae889-b4f9-4446-92a5-cc8715472411","Type":"ContainerStarted","Data":"886b9be72fb994bd3b3b5872710f8b16d119efface634a60e12b7919d947646a"}
Sep 19 10:43:56.835866 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:56.835816    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wzqdr" podStartSLOduration=2.404709507 podStartE2EDuration="6.835801143s" totalImagesPullingTime="4.431091636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:43:56.834807668 +0000 UTC m=+481.493552829" watchObservedRunningTime="2026-09-19 10:43:56.835801143 +0000 UTC m=+481.494546303"
Sep 19 10:43:57.618067 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:57.618025    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:57.716861 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:57.716813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:43:59.787568 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:43:59.787537    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:44:01.794900 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.794868    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"]
Sep 19 10:44:01.795324 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.795036    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l" podUID="0180753f-d097-49dc-80ed-e8c540e9d7cd" containerName="manager" containerID="cri-o://939b88c83b92da9e9c2a934d478fc8f0ba240d6d3666a8ffdb57d3a392602b2a" gracePeriod=2
Sep 19 10:44:01.807400 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.807372    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"]
Sep 19 10:44:01.826333 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:44:01.826254    2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0180753f-d097-49dc-80ed-e8c540e9d7cd\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-ccz6l\": pods \"kuadrant-operator-controller-manager-76f8f686bf-ccz6l\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:44:01.829526 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.829465    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"]
Sep 19 10:44:01.830288 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.829893    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0180753f-d097-49dc-80ed-e8c540e9d7cd" containerName="manager"
Sep 19 10:44:01.830288 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.829912    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0180753f-d097-49dc-80ed-e8c540e9d7cd" containerName="manager"
Sep 19 10:44:01.830288 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.830002    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0180753f-d097-49dc-80ed-e8c540e9d7cd" containerName="manager"
Sep 19 10:44:01.833003 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.832983    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"]
Sep 19 10:44:01.833173 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.833153    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n" podUID="01b44ae0-9e72-4493-b89a-8c7a1d5db9da" containerName="manager" containerID="cri-o://2716441a0846f932d0bd63691b590bb4de6e61fdef1f48071fb729f17b3ddff4" gracePeriod=2
Sep 19 10:44:01.833263 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.833159    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"
Sep 19 10:44:01.840680 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.840650    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"]
Sep 19 10:44:01.841960 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:44:01.841914    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ccz6l\" is forbidden: User \"system:node:ip-10-0-129-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-220.ec2.internal' and this object" podUID="0180753f-d097-49dc-80ed-e8c540e9d7cd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:44:01.843709 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:44:01.843667    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-wxr7n\" is forbidden: User \"system:node:ip-10-0-129-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-220.ec2.internal' and this object" podUID="01b44ae0-9e72-4493-b89a-8c7a1d5db9da" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"
Sep 19 10:44:01.849480 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.849456    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"]
Sep 19 10:44:01.857900 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.857874    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tm6rs"]
Sep 19 10:44:01.858145 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.858133    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b44ae0-9e72-4493-b89a-8c7a1d5db9da" containerName="manager"
Sep 19 10:44:01.858145 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.858145    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b44ae0-9e72-4493-b89a-8c7a1d5db9da" containerName="manager"
Sep 19 10:44:01.858276 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.858200    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b44ae0-9e72-4493-b89a-8c7a1d5db9da" containerName="manager"
Sep 19 10:44:01.861105 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.861085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tm6rs"
Sep 19 10:44:01.868694 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:44:01.868656    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ccz6l\" is forbidden: User \"system:node:ip-10-0-129-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-220.ec2.internal' and this object" podUID="0180753f-d097-49dc-80ed-e8c540e9d7cd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:44:01.869724 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.869622    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tm6rs"]
Sep 19 10:44:01.965234 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.965198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb669\" (UniqueName: \"kubernetes.io/projected/7ffdf181-a443-40c0-89d2-784c52d59871-kube-api-access-mb669\") pod \"limitador-operator-controller-manager-74495c69f8-tm6rs\" (UID: \"7ffdf181-a443-40c0-89d2-784c52d59871\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tm6rs"
Sep 19 10:44:01.965368 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.965299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/232835b8-af2a-4a28-88e5-d7f5119d1524-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4rztb\" (UID: \"232835b8-af2a-4a28-88e5-d7f5119d1524\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"
Sep 19 10:44:01.965368 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:01.965331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z26bl\" (UniqueName: \"kubernetes.io/projected/232835b8-af2a-4a28-88e5-d7f5119d1524-kube-api-access-z26bl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4rztb\" (UID: \"232835b8-af2a-4a28-88e5-d7f5119d1524\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"
Sep 19 10:44:02.066596 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.066551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/232835b8-af2a-4a28-88e5-d7f5119d1524-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4rztb\" (UID: \"232835b8-af2a-4a28-88e5-d7f5119d1524\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"
Sep 19 10:44:02.066717 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.066614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z26bl\" (UniqueName: \"kubernetes.io/projected/232835b8-af2a-4a28-88e5-d7f5119d1524-kube-api-access-z26bl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4rztb\" (UID: \"232835b8-af2a-4a28-88e5-d7f5119d1524\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"
Sep 19 10:44:02.066717 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.066686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb669\" (UniqueName: \"kubernetes.io/projected/7ffdf181-a443-40c0-89d2-784c52d59871-kube-api-access-mb669\") pod \"limitador-operator-controller-manager-74495c69f8-tm6rs\" (UID: \"7ffdf181-a443-40c0-89d2-784c52d59871\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tm6rs"
Sep 19 10:44:02.067012 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.066970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/232835b8-af2a-4a28-88e5-d7f5119d1524-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4rztb\" (UID: \"232835b8-af2a-4a28-88e5-d7f5119d1524\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"
Sep 19 10:44:02.075770 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.075732    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:44:02.078172 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.078155    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"
Sep 19 10:44:02.079798 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.079741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb669\" (UniqueName: \"kubernetes.io/projected/7ffdf181-a443-40c0-89d2-784c52d59871-kube-api-access-mb669\") pod \"limitador-operator-controller-manager-74495c69f8-tm6rs\" (UID: \"7ffdf181-a443-40c0-89d2-784c52d59871\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tm6rs"
Sep 19 10:44:02.080690 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.080670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z26bl\" (UniqueName: \"kubernetes.io/projected/232835b8-af2a-4a28-88e5-d7f5119d1524-kube-api-access-z26bl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4rztb\" (UID: \"232835b8-af2a-4a28-88e5-d7f5119d1524\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"
Sep 19 10:44:02.167166 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.167134    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0180753f-d097-49dc-80ed-e8c540e9d7cd-extensions-socket-volume\") pod \"0180753f-d097-49dc-80ed-e8c540e9d7cd\" (UID: \"0180753f-d097-49dc-80ed-e8c540e9d7cd\") "
Sep 19 10:44:02.167353 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.167184    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvsln\" (UniqueName: \"kubernetes.io/projected/0180753f-d097-49dc-80ed-e8c540e9d7cd-kube-api-access-hvsln\") pod \"0180753f-d097-49dc-80ed-e8c540e9d7cd\" (UID: \"0180753f-d097-49dc-80ed-e8c540e9d7cd\") "
Sep 19 10:44:02.167353 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.167228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgz94\" (UniqueName: \"kubernetes.io/projected/01b44ae0-9e72-4493-b89a-8c7a1d5db9da-kube-api-access-hgz94\") pod \"01b44ae0-9e72-4493-b89a-8c7a1d5db9da\" (UID: \"01b44ae0-9e72-4493-b89a-8c7a1d5db9da\") "
Sep 19 10:44:02.167577 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.167556    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0180753f-d097-49dc-80ed-e8c540e9d7cd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0180753f-d097-49dc-80ed-e8c540e9d7cd" (UID: "0180753f-d097-49dc-80ed-e8c540e9d7cd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:44:02.169549 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.169525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0180753f-d097-49dc-80ed-e8c540e9d7cd-kube-api-access-hvsln" (OuterVolumeSpecName: "kube-api-access-hvsln") pod "0180753f-d097-49dc-80ed-e8c540e9d7cd" (UID: "0180753f-d097-49dc-80ed-e8c540e9d7cd"). InnerVolumeSpecName "kube-api-access-hvsln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:44:02.169610 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.169571    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b44ae0-9e72-4493-b89a-8c7a1d5db9da-kube-api-access-hgz94" (OuterVolumeSpecName: "kube-api-access-hgz94") pod "01b44ae0-9e72-4493-b89a-8c7a1d5db9da" (UID: "01b44ae0-9e72-4493-b89a-8c7a1d5db9da"). InnerVolumeSpecName "kube-api-access-hgz94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:44:02.253943 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.253904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"
Sep 19 10:44:02.260803 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.260778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tm6rs"
Sep 19 10:44:02.268447 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.268423    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0180753f-d097-49dc-80ed-e8c540e9d7cd-extensions-socket-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:44:02.268447 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.268448    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvsln\" (UniqueName: \"kubernetes.io/projected/0180753f-d097-49dc-80ed-e8c540e9d7cd-kube-api-access-hvsln\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:44:02.268563 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.268461    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgz94\" (UniqueName: \"kubernetes.io/projected/01b44ae0-9e72-4493-b89a-8c7a1d5db9da-kube-api-access-hgz94\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:44:02.538805 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.538747    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tm6rs"]
Sep 19 10:44:02.540670 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.540644    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"]
Sep 19 10:44:02.542150 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:44:02.542126    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffdf181_a443_40c0_89d2_784c52d59871.slice/crio-8944d980c8e70e4902504f6dbd8a9388b4e95bb5238a8ffef79fb1f0678a844b WatchSource:0}: Error finding container 8944d980c8e70e4902504f6dbd8a9388b4e95bb5238a8ffef79fb1f0678a844b: Status 404 returned error can't find the container with id 8944d980c8e70e4902504f6dbd8a9388b4e95bb5238a8ffef79fb1f0678a844b
Sep 19 10:44:02.542769 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:44:02.542740    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232835b8_af2a_4a28_88e5_d7f5119d1524.slice/crio-cdfcb2745967ffb9b176fb8d65702f7ac03bfbdf70cd2da4c183df420b291742 WatchSource:0}: Error finding container cdfcb2745967ffb9b176fb8d65702f7ac03bfbdf70cd2da4c183df420b291742: Status 404 returned error can't find the container with id cdfcb2745967ffb9b176fb8d65702f7ac03bfbdf70cd2da4c183df420b291742
Sep 19 10:44:02.854008 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.853975    2580 generic.go:356] "Generic (PLEG): container finished" podID="0180753f-d097-49dc-80ed-e8c540e9d7cd" containerID="939b88c83b92da9e9c2a934d478fc8f0ba240d6d3666a8ffdb57d3a392602b2a" exitCode=0
Sep 19 10:44:02.854481 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.854047    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccz6l"
Sep 19 10:44:02.854481 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.854061    2580 scope.go:117] "RemoveContainer" containerID="939b88c83b92da9e9c2a934d478fc8f0ba240d6d3666a8ffdb57d3a392602b2a"
Sep 19 10:44:02.856136 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.856050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tm6rs" event={"ID":"7ffdf181-a443-40c0-89d2-784c52d59871","Type":"ContainerStarted","Data":"6f7558c28f0b28f7b743e0d966feb44ef9e567ab2ca2c7dbe71c6825c6aac5d2"}
Sep 19 10:44:02.856254 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.856156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tm6rs" event={"ID":"7ffdf181-a443-40c0-89d2-784c52d59871","Type":"ContainerStarted","Data":"8944d980c8e70e4902504f6dbd8a9388b4e95bb5238a8ffef79fb1f0678a844b"}
Sep 19 10:44:02.856254 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.856188    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tm6rs"
Sep 19 10:44:02.858188 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.858158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb" event={"ID":"232835b8-af2a-4a28-88e5-d7f5119d1524","Type":"ContainerStarted","Data":"8efaffbf508d4e19102b36d95750459937f847094bd12749678cd3c11cbcf699"}
Sep 19 10:44:02.858310 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.858209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb" event={"ID":"232835b8-af2a-4a28-88e5-d7f5119d1524","Type":"ContainerStarted","Data":"cdfcb2745967ffb9b176fb8d65702f7ac03bfbdf70cd2da4c183df420b291742"}
Sep 19 10:44:02.858310 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.858238    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"
Sep 19 10:44:02.859947 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.859920    2580 generic.go:356] "Generic (PLEG): container finished" podID="01b44ae0-9e72-4493-b89a-8c7a1d5db9da" containerID="2716441a0846f932d0bd63691b590bb4de6e61fdef1f48071fb729f17b3ddff4" exitCode=0
Sep 19 10:44:02.860052 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.859972    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wxr7n"
Sep 19 10:44:02.865191 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.865169    2580 scope.go:117] "RemoveContainer" containerID="939b88c83b92da9e9c2a934d478fc8f0ba240d6d3666a8ffdb57d3a392602b2a"
Sep 19 10:44:02.865511 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:44:02.865486    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939b88c83b92da9e9c2a934d478fc8f0ba240d6d3666a8ffdb57d3a392602b2a\": container with ID starting with 939b88c83b92da9e9c2a934d478fc8f0ba240d6d3666a8ffdb57d3a392602b2a not found: ID does not exist" containerID="939b88c83b92da9e9c2a934d478fc8f0ba240d6d3666a8ffdb57d3a392602b2a"
Sep 19 10:44:02.865571 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.865525    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939b88c83b92da9e9c2a934d478fc8f0ba240d6d3666a8ffdb57d3a392602b2a"} err="failed to get container status \"939b88c83b92da9e9c2a934d478fc8f0ba240d6d3666a8ffdb57d3a392602b2a\": rpc error: code = NotFound desc = could not find container \"939b88c83b92da9e9c2a934d478fc8f0ba240d6d3666a8ffdb57d3a392602b2a\": container with ID starting with 939b88c83b92da9e9c2a934d478fc8f0ba240d6d3666a8ffdb57d3a392602b2a not found: ID does not exist"
Sep 19 10:44:02.865571 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.865551    2580 scope.go:117] "RemoveContainer" containerID="2716441a0846f932d0bd63691b590bb4de6e61fdef1f48071fb729f17b3ddff4"
Sep 19 10:44:02.874749 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.874721    2580 scope.go:117] "RemoveContainer" containerID="2716441a0846f932d0bd63691b590bb4de6e61fdef1f48071fb729f17b3ddff4"
Sep 19 10:44:02.875139 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:44:02.875114    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2716441a0846f932d0bd63691b590bb4de6e61fdef1f48071fb729f17b3ddff4\": container with ID starting with 2716441a0846f932d0bd63691b590bb4de6e61fdef1f48071fb729f17b3ddff4 not found: ID does not exist" containerID="2716441a0846f932d0bd63691b590bb4de6e61fdef1f48071fb729f17b3ddff4"
Sep 19 10:44:02.875245 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.875143    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2716441a0846f932d0bd63691b590bb4de6e61fdef1f48071fb729f17b3ddff4"} err="failed to get container status \"2716441a0846f932d0bd63691b590bb4de6e61fdef1f48071fb729f17b3ddff4\": rpc error: code = NotFound desc = could not find container \"2716441a0846f932d0bd63691b590bb4de6e61fdef1f48071fb729f17b3ddff4\": container with ID starting with 2716441a0846f932d0bd63691b590bb4de6e61fdef1f48071fb729f17b3ddff4 not found: ID does not exist"
Sep 19 10:44:02.877099 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.877061    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tm6rs" podStartSLOduration=1.87704896 podStartE2EDuration="1.87704896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:44:02.875815541 +0000 UTC m=+487.534560701" watchObservedRunningTime="2026-09-19 10:44:02.87704896 +0000 UTC m=+487.535794114"
Sep 19 10:44:02.894488 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:02.894412    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb" podStartSLOduration=1.894393759 podStartE2EDuration="1.894393759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:44:02.893452876 +0000 UTC m=+487.552198035" watchObservedRunningTime="2026-09-19 10:44:02.894393759 +0000 UTC m=+487.553138919"
Sep 19 10:44:03.856531 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:03.856497    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0180753f-d097-49dc-80ed-e8c540e9d7cd" path="/var/lib/kubelet/pods/0180753f-d097-49dc-80ed-e8c540e9d7cd/volumes"
Sep 19 10:44:03.856906 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:03.856850    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b44ae0-9e72-4493-b89a-8c7a1d5db9da" path="/var/lib/kubelet/pods/01b44ae0-9e72-4493-b89a-8c7a1d5db9da/volumes"
Sep 19 10:44:04.487485 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:04.487446    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:04.588306 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:04.588270    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:04.848808 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:04.848706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:04.952533 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:04.952500    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:05.846479 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:05.846450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:05.946890 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:05.946864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:08.445450 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:08.445414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:08.546503 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:08.546468    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:11.445984 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:11.445943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:11.492824 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:11.492788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:13.868659 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:13.868631    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tm6rs"
Sep 19 10:44:13.869053 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:13.868701    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"
Sep 19 10:44:17.188016 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.187978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:17.288525 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.288488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:17.344025 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.343988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:17.437422 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.437389    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"]
Sep 19 10:44:17.437620 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.437594    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb" podUID="232835b8-af2a-4a28-88e5-d7f5119d1524" containerName="manager" containerID="cri-o://8efaffbf508d4e19102b36d95750459937f847094bd12749678cd3c11cbcf699" gracePeriod=10
Sep 19 10:44:17.677633 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.677607    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"
Sep 19 10:44:17.687940 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.687912    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"]
Sep 19 10:44:17.688290 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.688242    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232835b8-af2a-4a28-88e5-d7f5119d1524" containerName="manager"
Sep 19 10:44:17.688290 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.688260    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="232835b8-af2a-4a28-88e5-d7f5119d1524" containerName="manager"
Sep 19 10:44:17.688421 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.688357    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="232835b8-af2a-4a28-88e5-d7f5119d1524" containerName="manager"
Sep 19 10:44:17.690500 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.690481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 10:44:17.705473 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.705450    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"]
Sep 19 10:44:17.787690 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.787648    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z26bl\" (UniqueName: \"kubernetes.io/projected/232835b8-af2a-4a28-88e5-d7f5119d1524-kube-api-access-z26bl\") pod \"232835b8-af2a-4a28-88e5-d7f5119d1524\" (UID: \"232835b8-af2a-4a28-88e5-d7f5119d1524\") "
Sep 19 10:44:17.787895 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.787708    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/232835b8-af2a-4a28-88e5-d7f5119d1524-extensions-socket-volume\") pod \"232835b8-af2a-4a28-88e5-d7f5119d1524\" (UID: \"232835b8-af2a-4a28-88e5-d7f5119d1524\") "
Sep 19 10:44:17.787895 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.787859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw4wb\" (UniqueName: \"kubernetes.io/projected/727bbfa8-e3c8-4f40-b05f-f31161f3b1ad-kube-api-access-pw4wb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c9z6z\" (UID: \"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 10:44:17.787979 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.787907    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/727bbfa8-e3c8-4f40-b05f-f31161f3b1ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c9z6z\" (UID: \"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 10:44:17.788106 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.788083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232835b8-af2a-4a28-88e5-d7f5119d1524-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "232835b8-af2a-4a28-88e5-d7f5119d1524" (UID: "232835b8-af2a-4a28-88e5-d7f5119d1524"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:44:17.789935 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.789912    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232835b8-af2a-4a28-88e5-d7f5119d1524-kube-api-access-z26bl" (OuterVolumeSpecName: "kube-api-access-z26bl") pod "232835b8-af2a-4a28-88e5-d7f5119d1524" (UID: "232835b8-af2a-4a28-88e5-d7f5119d1524"). InnerVolumeSpecName "kube-api-access-z26bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:44:17.888291 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.888262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4wb\" (UniqueName: \"kubernetes.io/projected/727bbfa8-e3c8-4f40-b05f-f31161f3b1ad-kube-api-access-pw4wb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c9z6z\" (UID: \"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 10:44:17.888494 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.888323    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/727bbfa8-e3c8-4f40-b05f-f31161f3b1ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c9z6z\" (UID: \"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 10:44:17.888494 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.888362    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/232835b8-af2a-4a28-88e5-d7f5119d1524-extensions-socket-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:44:17.888494 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.888372    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z26bl\" (UniqueName: \"kubernetes.io/projected/232835b8-af2a-4a28-88e5-d7f5119d1524-kube-api-access-z26bl\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:44:17.888709 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.888692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/727bbfa8-e3c8-4f40-b05f-f31161f3b1ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c9z6z\" (UID: \"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 10:44:17.897185 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.897159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4wb\" (UniqueName: \"kubernetes.io/projected/727bbfa8-e3c8-4f40-b05f-f31161f3b1ad-kube-api-access-pw4wb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c9z6z\" (UID: \"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 10:44:17.932516 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.932485    2580 generic.go:356] "Generic (PLEG): container finished" podID="232835b8-af2a-4a28-88e5-d7f5119d1524" containerID="8efaffbf508d4e19102b36d95750459937f847094bd12749678cd3c11cbcf699" exitCode=0
Sep 19 10:44:17.932675 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.932527    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"
Sep 19 10:44:17.932675 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.932563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb" event={"ID":"232835b8-af2a-4a28-88e5-d7f5119d1524","Type":"ContainerDied","Data":"8efaffbf508d4e19102b36d95750459937f847094bd12749678cd3c11cbcf699"}
Sep 19 10:44:17.932675 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.932623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb" event={"ID":"232835b8-af2a-4a28-88e5-d7f5119d1524","Type":"ContainerDied","Data":"cdfcb2745967ffb9b176fb8d65702f7ac03bfbdf70cd2da4c183df420b291742"}
Sep 19 10:44:17.932675 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.932654    2580 scope.go:117] "RemoveContainer" containerID="8efaffbf508d4e19102b36d95750459937f847094bd12749678cd3c11cbcf699"
Sep 19 10:44:17.941944 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.941876    2580 scope.go:117] "RemoveContainer" containerID="8efaffbf508d4e19102b36d95750459937f847094bd12749678cd3c11cbcf699"
Sep 19 10:44:17.942192 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:44:17.942172    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8efaffbf508d4e19102b36d95750459937f847094bd12749678cd3c11cbcf699\": container with ID starting with 8efaffbf508d4e19102b36d95750459937f847094bd12749678cd3c11cbcf699 not found: ID does not exist" containerID="8efaffbf508d4e19102b36d95750459937f847094bd12749678cd3c11cbcf699"
Sep 19 10:44:17.942232 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.942201    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efaffbf508d4e19102b36d95750459937f847094bd12749678cd3c11cbcf699"} err="failed to get container status \"8efaffbf508d4e19102b36d95750459937f847094bd12749678cd3c11cbcf699\": rpc error: code = NotFound desc = could not find container \"8efaffbf508d4e19102b36d95750459937f847094bd12749678cd3c11cbcf699\": container with ID starting with 8efaffbf508d4e19102b36d95750459937f847094bd12749678cd3c11cbcf699 not found: ID does not exist"
Sep 19 10:44:17.948592 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.948567    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"]
Sep 19 10:44:17.954560 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:17.954536    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4rztb"]
Sep 19 10:44:18.000357 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:18.000323    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 10:44:18.311392 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:18.311367    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"]
Sep 19 10:44:18.313930 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:44:18.313906    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727bbfa8_e3c8_4f40_b05f_f31161f3b1ad.slice/crio-9c23c8c2d038ce27b76c60775f4cf550e961380b3742647e49dac10b46bb31a4 WatchSource:0}: Error finding container 9c23c8c2d038ce27b76c60775f4cf550e961380b3742647e49dac10b46bb31a4: Status 404 returned error can't find the container with id 9c23c8c2d038ce27b76c60775f4cf550e961380b3742647e49dac10b46bb31a4
Sep 19 10:44:18.940139 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:18.940097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z" event={"ID":"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad","Type":"ContainerStarted","Data":"3ba3c1413698a10426615ca08fdfd54a10e05d268761f6dc895f30471734780a"}
Sep 19 10:44:18.940339 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:18.940150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z" event={"ID":"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad","Type":"ContainerStarted","Data":"9c23c8c2d038ce27b76c60775f4cf550e961380b3742647e49dac10b46bb31a4"}
Sep 19 10:44:18.940339 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:18.940179    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 10:44:18.970203 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:18.970150    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z" podStartSLOduration=1.970131811 podStartE2EDuration="1.970131811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:44:18.968919103 +0000 UTC m=+503.627664264" watchObservedRunningTime="2026-09-19 10:44:18.970131811 +0000 UTC m=+503.628876972"
Sep 19 10:44:19.858471 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:19.858429    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="232835b8-af2a-4a28-88e5-d7f5119d1524" path="/var/lib/kubelet/pods/232835b8-af2a-4a28-88e5-d7f5119d1524/volumes"
Sep 19 10:44:29.947779 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:29.947721    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 10:44:31.716060 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:31.716019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:31.765936 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:31.765902    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:33.739436 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:33.739377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:33.969472 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:33.969416    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:34.557103 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:34.557050    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:34.658244 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:34.658211    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:50.189535 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.189431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:50.291830 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.291740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:50.344070 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.344035    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:50.581679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.581587    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:44:50.585120 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.585096    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-4th8x"
Sep 19 10:44:50.587153 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.587130    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 19 10:44:50.591777 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.591723    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:44:50.621382 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.621351    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:44:50.641325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.641290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/55daafdd-2998-45d4-9323-8b438b474d5b-config-file\") pod \"limitador-limitador-84994f7ffc-4th8x\" (UID: \"55daafdd-2998-45d4-9323-8b438b474d5b\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4th8x"
Sep 19 10:44:50.641480 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.641341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm9d4\" (UniqueName: \"kubernetes.io/projected/55daafdd-2998-45d4-9323-8b438b474d5b-kube-api-access-vm9d4\") pod \"limitador-limitador-84994f7ffc-4th8x\" (UID: \"55daafdd-2998-45d4-9323-8b438b474d5b\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4th8x"
Sep 19 10:44:50.742548 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.742507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/55daafdd-2998-45d4-9323-8b438b474d5b-config-file\") pod \"limitador-limitador-84994f7ffc-4th8x\" (UID: \"55daafdd-2998-45d4-9323-8b438b474d5b\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4th8x"
Sep 19 10:44:50.742749 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.742579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm9d4\" (UniqueName: \"kubernetes.io/projected/55daafdd-2998-45d4-9323-8b438b474d5b-kube-api-access-vm9d4\") pod \"limitador-limitador-84994f7ffc-4th8x\" (UID: \"55daafdd-2998-45d4-9323-8b438b474d5b\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4th8x"
Sep 19 10:44:50.743169 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.743148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/55daafdd-2998-45d4-9323-8b438b474d5b-config-file\") pod \"limitador-limitador-84994f7ffc-4th8x\" (UID: \"55daafdd-2998-45d4-9323-8b438b474d5b\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4th8x"
Sep 19 10:44:50.750329 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.750307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm9d4\" (UniqueName: \"kubernetes.io/projected/55daafdd-2998-45d4-9323-8b438b474d5b-kube-api-access-vm9d4\") pod \"limitador-limitador-84994f7ffc-4th8x\" (UID: \"55daafdd-2998-45d4-9323-8b438b474d5b\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4th8x"
Sep 19 10:44:50.898089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:50.897987    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-4th8x"
Sep 19 10:44:51.021891 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:51.021857    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:51.119382 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:51.119351    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:44:51.166107 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:51.165997    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:44:51.168816 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:44:51.168781    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55daafdd_2998_45d4_9323_8b438b474d5b.slice/crio-c00cbcab1712cf873abf2f12c01082c0a025d66da0f9c296cee0756eb748790f WatchSource:0}: Error finding container c00cbcab1712cf873abf2f12c01082c0a025d66da0f9c296cee0756eb748790f: Status 404 returned error can't find the container with id c00cbcab1712cf873abf2f12c01082c0a025d66da0f9c296cee0756eb748790f
Sep 19 10:44:52.093907 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:52.093862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-4th8x" event={"ID":"55daafdd-2998-45d4-9323-8b438b474d5b","Type":"ContainerStarted","Data":"9a88adc536d289752288e2a32cdf06fce06e194b6bf1aa6dcc77dfcd620ca38d"}
Sep 19 10:44:52.094251 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:52.093925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-4th8x" event={"ID":"55daafdd-2998-45d4-9323-8b438b474d5b","Type":"ContainerStarted","Data":"c00cbcab1712cf873abf2f12c01082c0a025d66da0f9c296cee0756eb748790f"}
Sep 19 10:44:52.094251 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:52.093950    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-4th8x"
Sep 19 10:44:52.109132 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:44:52.109072    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-4th8x" podStartSLOduration=1.255555153 podStartE2EDuration="2.109051586s" totalImagesPullingTime="853.496433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:44:52.107783432 +0000 UTC m=+536.766528593" watchObservedRunningTime="2026-09-19 10:44:52.109051586 +0000 UTC m=+536.767796749"
Sep 19 10:45:03.099615 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:03.099584    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-4th8x"
Sep 19 10:45:06.969135 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:06.969100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:45:07.068500 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:07.068457    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:45:07.406278 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:07.406187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:45:20.068520 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:20.068479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:45:20.680712 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:20.680668    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:45:26.372846 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:26.372810    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:45:26.445146 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:26.445109    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:45:26.507667 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:26.507626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:45:26.575308 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:26.575273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:45:29.752825 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:29.752785    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:45:29.859052 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:29.859017    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:45:55.785300 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:55.785270    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 10:45:55.799436 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:45:55.799403    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 10:46:01.410387 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:01.410336    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:01.512018 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:01.511971    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:02.816859 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:02.816827    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:02.916974 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:02.916938    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:03.017213 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:03.017164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:03.116988 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:03.116903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:03.378788 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:03.378668    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:03.473314 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:03.473274    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:03.597867 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:03.597822    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:46:05.834255 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:05.834216    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:05.887672 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:05.887639    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:13.026176 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:13.026139    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:13.119154 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:13.119117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:16.920464 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:16.920382    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"]
Sep 19 10:46:16.923564 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:16.923543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:16.926652 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:16.926511    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 19 10:46:16.926652 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:16.926526    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 19 10:46:16.926652 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:16.926515    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 19 10:46:16.926652 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:16.926560    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-prjcd\""
Sep 19 10:46:16.932062 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:16.931567    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"]
Sep 19 10:46:16.946697 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:16.946669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:17.010618 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.010590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.010839 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.010640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvsvr\" (UniqueName: \"kubernetes.io/projected/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-kube-api-access-hvsvr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.010839 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.010667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.010839 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.010689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.010839 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.010720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.010839 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.010735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.111877 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.111842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.111877 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.111879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.112100 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.112029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.112146 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.112098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvsvr\" (UniqueName: \"kubernetes.io/projected/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-kube-api-access-hvsvr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.112189 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.112141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.112189 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.112175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.112299 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.112282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.112423 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.112406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.112540 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.112511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.114311 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.114291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.114630 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.114609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.119478 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.119454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvsvr\" (UniqueName: \"kubernetes.io/projected/d3cd7942-80aa-4ee4-86e9-8a5340c414c4-kube-api-access-hvsvr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc\" (UID: \"d3cd7942-80aa-4ee4-86e9-8a5340c414c4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.236266 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.236229    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:17.508516 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.508437    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"]
Sep 19 10:46:17.511321 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.511292    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:46:17.511733 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:46:17.511712    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cd7942_80aa_4ee4_86e9_8a5340c414c4.slice/crio-311cd2bf19ba57fd7011c0359f1218a521a22f87911471689e263592ab49c722 WatchSource:0}: Error finding container 311cd2bf19ba57fd7011c0359f1218a521a22f87911471689e263592ab49c722: Status 404 returned error can't find the container with id 311cd2bf19ba57fd7011c0359f1218a521a22f87911471689e263592ab49c722
Sep 19 10:46:17.513074 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:17.513058    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 10:46:18.474020 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:18.473976    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc" event={"ID":"d3cd7942-80aa-4ee4-86e9-8a5340c414c4","Type":"ContainerStarted","Data":"311cd2bf19ba57fd7011c0359f1218a521a22f87911471689e263592ab49c722"}
Sep 19 10:46:19.480075 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:19.480038    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc" event={"ID":"d3cd7942-80aa-4ee4-86e9-8a5340c414c4","Type":"ContainerStarted","Data":"77047c29daa6f4b4571b7912e5e07d198c823ae27a3bbbf3d770821a61d898a2"}
Sep 19 10:46:19.495740 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:19.495688    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc" podStartSLOduration=2.364423775 podStartE2EDuration="3.495671619s" totalImagesPullingTime="1.131247844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:46:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:46:19.494121007 +0000 UTC m=+624.152866169" watchObservedRunningTime="2026-09-19 10:46:19.495671619 +0000 UTC m=+624.154416778"
Sep 19 10:46:22.937670 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:22.937628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:23.624098 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:23.624063    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:46:27.237388 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:27.237352    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:27.237956 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:27.237451    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:27.250127 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:27.250096    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:27.526572 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:27.526491    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc"
Sep 19 10:46:29.118220 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:29.118186    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:30.008232 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:30.008193    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:46:35.209807 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:35.209740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:37.262388 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:37.262355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:51.046313 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:51.046274    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:52.215387 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:52.215352    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:46:56.709542 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.709508    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"]
Sep 19 10:46:56.715644 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.714262    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.716953 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.716928    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 19 10:46:56.721339 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.721315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"]
Sep 19 10:46:56.734652 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.734610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d16400b-68ba-47f6-9939-2e798850814e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.734820 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.734676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d16400b-68ba-47f6-9939-2e798850814e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.734820 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.734716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d16400b-68ba-47f6-9939-2e798850814e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.734820 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.734808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d16400b-68ba-47f6-9939-2e798850814e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.734951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.734842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrscg\" (UniqueName: \"kubernetes.io/projected/6d16400b-68ba-47f6-9939-2e798850814e-kube-api-access-rrscg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.734951 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.734877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d16400b-68ba-47f6-9939-2e798850814e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.818060 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.818024    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:46:56.836023 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.835980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d16400b-68ba-47f6-9939-2e798850814e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.836231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.836040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrscg\" (UniqueName: \"kubernetes.io/projected/6d16400b-68ba-47f6-9939-2e798850814e-kube-api-access-rrscg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.836231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.836074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d16400b-68ba-47f6-9939-2e798850814e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.836231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.836127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d16400b-68ba-47f6-9939-2e798850814e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.836231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.836156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d16400b-68ba-47f6-9939-2e798850814e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.836231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.836195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d16400b-68ba-47f6-9939-2e798850814e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.836634 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.836608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d16400b-68ba-47f6-9939-2e798850814e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.836634 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.836621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d16400b-68ba-47f6-9939-2e798850814e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.836813 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.836679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d16400b-68ba-47f6-9939-2e798850814e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.838881 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.838862    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d16400b-68ba-47f6-9939-2e798850814e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.838944 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.838886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d16400b-68ba-47f6-9939-2e798850814e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:56.843449 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:56.843425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrscg\" (UniqueName: \"kubernetes.io/projected/6d16400b-68ba-47f6-9939-2e798850814e-kube-api-access-rrscg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t\" (UID: \"6d16400b-68ba-47f6-9939-2e798850814e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:57.026470 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:57.026352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:46:57.302092 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:57.301877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"]
Sep 19 10:46:57.304624 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:46:57.304597    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d16400b_68ba_47f6_9939_2e798850814e.slice/crio-189db26c72ac4ea80d3985d9f66d75c89929c9d028ec21dd6cc251b666441b7e WatchSource:0}: Error finding container 189db26c72ac4ea80d3985d9f66d75c89929c9d028ec21dd6cc251b666441b7e: Status 404 returned error can't find the container with id 189db26c72ac4ea80d3985d9f66d75c89929c9d028ec21dd6cc251b666441b7e
Sep 19 10:46:57.645979 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:57.645884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t" event={"ID":"6d16400b-68ba-47f6-9939-2e798850814e","Type":"ContainerStarted","Data":"8474bed48b42ebe335b655a7be8684a85f087c55af2ab00d21f9d1e63038b825"}
Sep 19 10:46:57.645979 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:57.645943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t" event={"ID":"6d16400b-68ba-47f6-9939-2e798850814e","Type":"ContainerStarted","Data":"189db26c72ac4ea80d3985d9f66d75c89929c9d028ec21dd6cc251b666441b7e"}
Sep 19 10:46:57.662447 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:57.662385    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t" podStartSLOduration=1.500050611 podStartE2EDuration="1.66236929s" totalImagesPullingTime="162.318679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:46:57.660392341 +0000 UTC m=+662.319137501" watchObservedRunningTime="2026-09-19 10:46:57.66236929 +0000 UTC m=+662.321114444"
Sep 19 10:46:58.406223 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:46:58.406185    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:47:00.829234 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:00.829196    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:47:03.515144 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:03.515110    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:47:07.027085 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:07.027042    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:47:07.027488 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:07.027116    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:47:07.040054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:07.040025    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:47:07.702594 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:07.702556    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t"
Sep 19 10:47:14.614037 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.614003    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"]
Sep 19 10:47:14.617636 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.617614    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.619994 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.619970    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 19 10:47:14.626395 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.626370    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"]
Sep 19 10:47:14.641162 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.641124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:47:14.690489 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.690454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37dbaf07-ee66-4cb3-b34c-50d772158ac8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.690657 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.690502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37dbaf07-ee66-4cb3-b34c-50d772158ac8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.690657 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.690529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjxg9\" (UniqueName: \"kubernetes.io/projected/37dbaf07-ee66-4cb3-b34c-50d772158ac8-kube-api-access-mjxg9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.690733 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.690649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37dbaf07-ee66-4cb3-b34c-50d772158ac8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.690733 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.690687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37dbaf07-ee66-4cb3-b34c-50d772158ac8-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.690733 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.690713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37dbaf07-ee66-4cb3-b34c-50d772158ac8-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.791450 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.791413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37dbaf07-ee66-4cb3-b34c-50d772158ac8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.791450 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.791461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37dbaf07-ee66-4cb3-b34c-50d772158ac8-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.791714 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.791483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37dbaf07-ee66-4cb3-b34c-50d772158ac8-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.791714 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.791514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37dbaf07-ee66-4cb3-b34c-50d772158ac8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.791714 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.791535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37dbaf07-ee66-4cb3-b34c-50d772158ac8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.791714 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.791558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjxg9\" (UniqueName: \"kubernetes.io/projected/37dbaf07-ee66-4cb3-b34c-50d772158ac8-kube-api-access-mjxg9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.791984 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.791958    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37dbaf07-ee66-4cb3-b34c-50d772158ac8-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.792051 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.791997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37dbaf07-ee66-4cb3-b34c-50d772158ac8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.792100 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.792043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37dbaf07-ee66-4cb3-b34c-50d772158ac8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.794001 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.793977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37dbaf07-ee66-4cb3-b34c-50d772158ac8-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.794174 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.794155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37dbaf07-ee66-4cb3-b34c-50d772158ac8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.798479 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.798453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjxg9\" (UniqueName: \"kubernetes.io/projected/37dbaf07-ee66-4cb3-b34c-50d772158ac8-kube-api-access-mjxg9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj\" (UID: \"37dbaf07-ee66-4cb3-b34c-50d772158ac8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:14.927931 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:14.927829    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:15.201349 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:15.201325    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"]
Sep 19 10:47:15.203898 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:47:15.203866    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37dbaf07_ee66_4cb3_b34c_50d772158ac8.slice/crio-e73bb26bfb13454073a5dac0ba3986675fdb0c75b31c89772c2c89611a11870e WatchSource:0}: Error finding container e73bb26bfb13454073a5dac0ba3986675fdb0c75b31c89772c2c89611a11870e: Status 404 returned error can't find the container with id e73bb26bfb13454073a5dac0ba3986675fdb0c75b31c89772c2c89611a11870e
Sep 19 10:47:15.736153 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:15.736120    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj" event={"ID":"37dbaf07-ee66-4cb3-b34c-50d772158ac8","Type":"ContainerStarted","Data":"d1aba147e9bfc9423fbdc761c95e74bec80c0033b2ec010535218224af2e74d5"}
Sep 19 10:47:15.736624 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:15.736163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj" event={"ID":"37dbaf07-ee66-4cb3-b34c-50d772158ac8","Type":"ContainerStarted","Data":"e73bb26bfb13454073a5dac0ba3986675fdb0c75b31c89772c2c89611a11870e"}
Sep 19 10:47:15.751780 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:15.751707    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj" podStartSLOduration=1.58962023 podStartE2EDuration="1.751691018s" totalImagesPullingTime="162.070788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:47:15.750732269 +0000 UTC m=+680.409477433" watchObservedRunningTime="2026-09-19 10:47:15.751691018 +0000 UTC m=+680.410436191"
Sep 19 10:47:16.816922 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:16.816890    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:47:24.928346 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:24.928300    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:24.928346 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:24.928349    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:24.941054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:24.941021    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:47:25.795352 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:47:25.795324    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj"
Sep 19 10:48:22.316223 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:22.316111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:22.415462 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:22.415430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:23.426904 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:23.426854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:23.535039 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:23.534998    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:38.957098 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:38.957061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:39.008642 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:39.008606    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:39.077073 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:39.077037    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:39.280283 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:39.280205    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:39.366775 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:39.366716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:39.430063 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:39.430021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:39.551108 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:39.551014    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:39.666420 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:39.666387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:39.801784 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:39.801662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:40.338399 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:40.338357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:40.442385 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:40.442346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:40.771277 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:40.771241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:40.896499 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:40.896462    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7980-26ae59"]
Sep 19 10:48:40.899661 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:40.899643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7980-26ae59"
Sep 19 10:48:40.901556 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:40.901527    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m27fx\""
Sep 19 10:48:40.906491 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:40.906384    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7980-26ae59"]
Sep 19 10:48:40.932896 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:40.932851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hnkt\" (UniqueName: \"kubernetes.io/projected/302f93c0-052f-4296-a7a7-347760a6b939-kube-api-access-4hnkt\") pod \"test-curl-7980-26ae59\" (UID: \"302f93c0-052f-4296-a7a7-347760a6b939\") " pod="opendatahub/test-curl-7980-26ae59"
Sep 19 10:48:41.033411 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:41.033312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnkt\" (UniqueName: \"kubernetes.io/projected/302f93c0-052f-4296-a7a7-347760a6b939-kube-api-access-4hnkt\") pod \"test-curl-7980-26ae59\" (UID: \"302f93c0-052f-4296-a7a7-347760a6b939\") " pod="opendatahub/test-curl-7980-26ae59"
Sep 19 10:48:41.041027 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:41.040994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hnkt\" (UniqueName: \"kubernetes.io/projected/302f93c0-052f-4296-a7a7-347760a6b939-kube-api-access-4hnkt\") pod \"test-curl-7980-26ae59\" (UID: \"302f93c0-052f-4296-a7a7-347760a6b939\") " pod="opendatahub/test-curl-7980-26ae59"
Sep 19 10:48:41.211085 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:41.211027    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7980-26ae59"
Sep 19 10:48:41.481919 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:41.481873    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7980-26ae59"]
Sep 19 10:48:41.484564 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:48:41.484531    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod302f93c0_052f_4296_a7a7_347760a6b939.slice/crio-bbaafe143e92d5fb011332631934e748f7cedbd61604e67c7c43a05a4c5d3512 WatchSource:0}: Error finding container bbaafe143e92d5fb011332631934e748f7cedbd61604e67c7c43a05a4c5d3512: Status 404 returned error can't find the container with id bbaafe143e92d5fb011332631934e748f7cedbd61604e67c7c43a05a4c5d3512
Sep 19 10:48:42.134120 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.134086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7980-26ae59" event={"ID":"302f93c0-052f-4296-a7a7-347760a6b939","Type":"ContainerStarted","Data":"8ebd0b302c0fed04135b08561aecfa16648537282fa6558b38238fe4fd7f037d"}
Sep 19 10:48:42.134120 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.134126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7980-26ae59" event={"ID":"302f93c0-052f-4296-a7a7-347760a6b939","Type":"ContainerStarted","Data":"bbaafe143e92d5fb011332631934e748f7cedbd61604e67c7c43a05a4c5d3512"}
Sep 19 10:48:42.147303 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.147250    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7980-26ae59" podStartSLOduration=1.571178974 podStartE2EDuration="2.147233552s" totalImagesPullingTime="576.054578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:48:42.146092182 +0000 UTC m=+766.804837344" watchObservedRunningTime="2026-09-19 10:48:42.147233552 +0000 UTC m=+766.805978727"
Sep 19 10:48:42.695130 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.695093    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7980-26ae59"]
Sep 19 10:48:42.697340 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.697308    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7980-26ae59"]
Sep 19 10:48:42.721611 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.721575    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:48:42.812096 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.811994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:42.820797 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.820728    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7980-7a335c"]
Sep 19 10:48:42.821166 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.821147    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="302f93c0-052f-4296-a7a7-347760a6b939" containerName="test-curl-7980-26ae59"
Sep 19 10:48:42.821255 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.821169    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="302f93c0-052f-4296-a7a7-347760a6b939" containerName="test-curl-7980-26ae59"
Sep 19 10:48:42.821255 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.821250    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="302f93c0-052f-4296-a7a7-347760a6b939" containerName="test-curl-7980-26ae59"
Sep 19 10:48:42.824827 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.824805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7980-7a335c"
Sep 19 10:48:42.830089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.830060    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7980-7a335c"]
Sep 19 10:48:42.846513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.846480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwmdp\" (UniqueName: \"kubernetes.io/projected/2ead14de-e934-4fc9-9398-2e3db095701b-kube-api-access-rwmdp\") pod \"test-curl-7980-7a335c\" (UID: \"2ead14de-e934-4fc9-9398-2e3db095701b\") " pod="opendatahub/test-curl-7980-7a335c"
Sep 19 10:48:42.902051 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.902016    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:42.947033 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.946986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwmdp\" (UniqueName: \"kubernetes.io/projected/2ead14de-e934-4fc9-9398-2e3db095701b-kube-api-access-rwmdp\") pod \"test-curl-7980-7a335c\" (UID: \"2ead14de-e934-4fc9-9398-2e3db095701b\") " pod="opendatahub/test-curl-7980-7a335c"
Sep 19 10:48:42.960143 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:42.960109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwmdp\" (UniqueName: \"kubernetes.io/projected/2ead14de-e934-4fc9-9398-2e3db095701b-kube-api-access-rwmdp\") pod \"test-curl-7980-7a335c\" (UID: \"2ead14de-e934-4fc9-9398-2e3db095701b\") " pod="opendatahub/test-curl-7980-7a335c"
Sep 19 10:48:43.136438 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:43.136346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7980-7a335c"
Sep 19 10:48:43.139132 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:48:43.139107    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7980-26ae59" container="test-curl-7980-26ae59"
Sep 19 10:48:43.406049 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:43.405843    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7980-7a335c"]
Sep 19 10:48:43.408382 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:48:43.408352    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ead14de_e934_4fc9_9398_2e3db095701b.slice/crio-fc056047cf6efd23e8ce515ec12c9313a83de5eb99a154473fef7a35f87cb873 WatchSource:0}: Error finding container fc056047cf6efd23e8ce515ec12c9313a83de5eb99a154473fef7a35f87cb873: Status 404 returned error can't find the container with id fc056047cf6efd23e8ce515ec12c9313a83de5eb99a154473fef7a35f87cb873
Sep 19 10:48:44.019399 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:44.019370    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:44.071279 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:44.071243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:44.145495 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:44.145460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7980-7a335c" event={"ID":"2ead14de-e934-4fc9-9398-2e3db095701b","Type":"ContainerStarted","Data":"d425465b2004391742355fc0391b106004062444bac280ea2de2d7fde48f653d"}
Sep 19 10:48:44.145495 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:44.145500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7980-7a335c" event={"ID":"2ead14de-e934-4fc9-9398-2e3db095701b","Type":"ContainerStarted","Data":"fc056047cf6efd23e8ce515ec12c9313a83de5eb99a154473fef7a35f87cb873"}
Sep 19 10:48:44.145729 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:44.145589    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7980-26ae59" podUID="302f93c0-052f-4296-a7a7-347760a6b939" containerName="test-curl-7980-26ae59" containerID="cri-o://8ebd0b302c0fed04135b08561aecfa16648537282fa6558b38238fe4fd7f037d" gracePeriod=2
Sep 19 10:48:44.160144 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:44.160079    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7980-7a335c" podStartSLOduration=2.160059043 podStartE2EDuration="2.160059043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:48:44.159231468 +0000 UTC m=+768.817976629" watchObservedRunningTime="2026-09-19 10:48:44.160059043 +0000 UTC m=+768.818804203"
Sep 19 10:48:44.534530 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:44.534493    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:44.666653 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:44.666559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:44.830002 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:44.829967    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7980-7a335c"]
Sep 19 10:48:44.833790 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:44.833738    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7980-7a335c"]
Sep 19 10:48:45.150476 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:48:45.150445    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7980-7a335c" container="test-curl-7980-7a335c"
Sep 19 10:48:45.336948 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:45.336913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:45.636825 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:45.636790    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:45.864062 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:45.864027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:46.034217 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:46.034177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:46.131096 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:46.131062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:46.155298 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:46.155259    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7980-7a335c" podUID="2ead14de-e934-4fc9-9398-2e3db095701b" containerName="test-curl-7980-7a335c" containerID="cri-o://d425465b2004391742355fc0391b106004062444bac280ea2de2d7fde48f653d" gracePeriod=2
Sep 19 10:48:46.233661 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:46.233021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:46.331682 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:46.331653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:46.333408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:46.333380    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7980-26ae59"
Sep 19 10:48:46.375742 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:46.375703    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hnkt\" (UniqueName: \"kubernetes.io/projected/302f93c0-052f-4296-a7a7-347760a6b939-kube-api-access-4hnkt\") pod \"302f93c0-052f-4296-a7a7-347760a6b939\" (UID: \"302f93c0-052f-4296-a7a7-347760a6b939\") "
Sep 19 10:48:46.378145 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:46.378119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302f93c0-052f-4296-a7a7-347760a6b939-kube-api-access-4hnkt" (OuterVolumeSpecName: "kube-api-access-4hnkt") pod "302f93c0-052f-4296-a7a7-347760a6b939" (UID: "302f93c0-052f-4296-a7a7-347760a6b939"). InnerVolumeSpecName "kube-api-access-4hnkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:48:46.476896 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:46.476854    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hnkt\" (UniqueName: \"kubernetes.io/projected/302f93c0-052f-4296-a7a7-347760a6b939-kube-api-access-4hnkt\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:48:46.923462 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:46.923428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:47.162106 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.162071    2580 generic.go:356] "Generic (PLEG): container finished" podID="302f93c0-052f-4296-a7a7-347760a6b939" containerID="8ebd0b302c0fed04135b08561aecfa16648537282fa6558b38238fe4fd7f037d" exitCode=137
Sep 19 10:48:47.162106 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.162120    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7980-26ae59"
Sep 19 10:48:47.162606 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.162173    2580 scope.go:117] "RemoveContainer" containerID="8ebd0b302c0fed04135b08561aecfa16648537282fa6558b38238fe4fd7f037d"
Sep 19 10:48:47.171580 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.171561    2580 scope.go:117] "RemoveContainer" containerID="8ebd0b302c0fed04135b08561aecfa16648537282fa6558b38238fe4fd7f037d"
Sep 19 10:48:47.171942 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:48:47.171921    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebd0b302c0fed04135b08561aecfa16648537282fa6558b38238fe4fd7f037d\": container with ID starting with 8ebd0b302c0fed04135b08561aecfa16648537282fa6558b38238fe4fd7f037d not found: ID does not exist" containerID="8ebd0b302c0fed04135b08561aecfa16648537282fa6558b38238fe4fd7f037d"
Sep 19 10:48:47.172011 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.171952    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebd0b302c0fed04135b08561aecfa16648537282fa6558b38238fe4fd7f037d"} err="failed to get container status \"8ebd0b302c0fed04135b08561aecfa16648537282fa6558b38238fe4fd7f037d\": rpc error: code = NotFound desc = could not find container \"8ebd0b302c0fed04135b08561aecfa16648537282fa6558b38238fe4fd7f037d\": container with ID starting with 8ebd0b302c0fed04135b08561aecfa16648537282fa6558b38238fe4fd7f037d not found: ID does not exist"
Sep 19 10:48:47.172716 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:48:47.172694    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7980-26ae59\" is forbidden: User \"system:node:ip-10-0-129-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-220.ec2.internal' and this object" podUID="302f93c0-052f-4296-a7a7-347760a6b939" pod="opendatahub/test-curl-7980-26ae59"
Sep 19 10:48:47.782641 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.782606    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"]
Sep 19 10:48:47.782984 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.782971    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ead14de-e934-4fc9-9398-2e3db095701b" containerName="test-curl-7980-7a335c"
Sep 19 10:48:47.783050 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.782986    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ead14de-e934-4fc9-9398-2e3db095701b" containerName="test-curl-7980-7a335c"
Sep 19 10:48:47.783050 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.783045    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ead14de-e934-4fc9-9398-2e3db095701b" containerName="test-curl-7980-7a335c"
Sep 19 10:48:47.787373 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.787348    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:47.789521 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.789495    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-1907c1-serving-cert\""
Sep 19 10:48:47.790123 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.790103    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g2pqv\""
Sep 19 10:48:47.790123 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.790117    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 19 10:48:47.790296 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.790123    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 19 10:48:47.790296 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.790187    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 10:48:47.796730 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.796465    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"]
Sep 19 10:48:47.812528 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:48:47.812486    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7980-26ae59\" is forbidden: User \"system:node:ip-10-0-129-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-220.ec2.internal' and this object" podUID="302f93c0-052f-4296-a7a7-347760a6b939" pod="opendatahub/test-curl-7980-26ae59"
Sep 19 10:48:47.817613 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.817578    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:47.863378 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.863333    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="302f93c0-052f-4296-a7a7-347760a6b939" path="/var/lib/kubelet/pods/302f93c0-052f-4296-a7a7-347760a6b939/volumes"
Sep 19 10:48:47.887492 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.887450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89353aaf-f742-434b-9b75-f6fde2b8babc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:47.887685 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.887496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:47.887685 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.887523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-metrics-certs\") pod \"maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:47.887685 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.887586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr4x9\" (UniqueName: \"kubernetes.io/projected/89353aaf-f742-434b-9b75-f6fde2b8babc-kube-api-access-tr4x9\") pod \"maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:47.921672 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.921634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:47.988297 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.988250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89353aaf-f742-434b-9b75-f6fde2b8babc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:47.988297 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.988303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:47.988641 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.988324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-metrics-certs\") pod \"maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:47.988641 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.988365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr4x9\" (UniqueName: \"kubernetes.io/projected/89353aaf-f742-434b-9b75-f6fde2b8babc-kube-api-access-tr4x9\") pod \"maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:47.988641 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:48:47.988436    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-serving-cert: secret "maas-api-e2e-worker-w0-1907c1-serving-cert" not found
Sep 19 10:48:47.988641 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:48:47.988518    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-maas-api-tls podName:89353aaf-f742-434b-9b75-f6fde2b8babc nodeName:}" failed. No retries permitted until 2026-09-19 10:48:48.488500105 +0000 UTC m=+773.147245245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-maas-api-tls") pod "maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98" (UID: "89353aaf-f742-434b-9b75-f6fde2b8babc") : secret "maas-api-e2e-worker-w0-1907c1-serving-cert" not found
Sep 19 10:48:47.989093 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.989070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89353aaf-f742-434b-9b75-f6fde2b8babc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:47.991365 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.991337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-metrics-certs\") pod \"maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:47.996943 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:47.996912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr4x9\" (UniqueName: \"kubernetes.io/projected/89353aaf-f742-434b-9b75-f6fde2b8babc-kube-api-access-tr4x9\") pod \"maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:48.144938 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.144849    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:48.248193 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.248157    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:48.345153 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.345049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:48.352075 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.352048    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7980-7a335c"
Sep 19 10:48:48.392285 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.392247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwmdp\" (UniqueName: \"kubernetes.io/projected/2ead14de-e934-4fc9-9398-2e3db095701b-kube-api-access-rwmdp\") pod \"2ead14de-e934-4fc9-9398-2e3db095701b\" (UID: \"2ead14de-e934-4fc9-9398-2e3db095701b\") "
Sep 19 10:48:48.394499 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.394471    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ead14de-e934-4fc9-9398-2e3db095701b-kube-api-access-rwmdp" (OuterVolumeSpecName: "kube-api-access-rwmdp") pod "2ead14de-e934-4fc9-9398-2e3db095701b" (UID: "2ead14de-e934-4fc9-9398-2e3db095701b"). InnerVolumeSpecName "kube-api-access-rwmdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:48:48.445246 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.445215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:48.493471 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.493435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:48.493688 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.493513    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwmdp\" (UniqueName: \"kubernetes.io/projected/2ead14de-e934-4fc9-9398-2e3db095701b-kube-api-access-rwmdp\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:48:48.496116 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.496093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:48.660854 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.658493    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"]
Sep 19 10:48:48.671055 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.671020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.673322 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.673294    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 19 10:48:48.673468 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.673299    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c152d119-openshift-default-dockercfg-wzn4w\""
Sep 19 10:48:48.679824 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.679789    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"]
Sep 19 10:48:48.694775 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.694690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-workload-certs\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.694971 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.694793    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e2ff36-97ab-440f-bcb3-cdfc65699add-istiod-ca-cert\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.694971 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.694836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4wg\" (UniqueName: \"kubernetes.io/projected/89e2ff36-97ab-440f-bcb3-cdfc65699add-kube-api-access-bx4wg\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.694971 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.694860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-workload-socket\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.694971 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.694882    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-credential-socket\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.694971 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.694953    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-data\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.695210 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.695036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-podinfo\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.695210 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.695062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-envoy\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.695210 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.695082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-token\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.699307 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.699221    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:48.701041 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.701016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:48.796186 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.796150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e2ff36-97ab-440f-bcb3-cdfc65699add-istiod-ca-cert\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.796368 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.796190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4wg\" (UniqueName: \"kubernetes.io/projected/89e2ff36-97ab-440f-bcb3-cdfc65699add-kube-api-access-bx4wg\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.796368 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.796211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-workload-socket\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.796480 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.796389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-credential-socket\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.796480 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.796438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-data\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.796578 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.796526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-podinfo\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.796578 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.796558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-envoy\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.796676 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.796585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-token\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.796676 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.796617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-workload-certs\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.796798 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.796586    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-workload-socket\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.796879 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.796854    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-credential-socket\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.797016 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.796991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-data\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.797124 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.797034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e2ff36-97ab-440f-bcb3-cdfc65699add-istiod-ca-cert\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.797124 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.797083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-workload-certs\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.798911 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.798893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-envoy\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.799155 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.799138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-podinfo\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.803211 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.803183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4wg\" (UniqueName: \"kubernetes.io/projected/89e2ff36-97ab-440f-bcb3-cdfc65699add-kube-api-access-bx4wg\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.803331 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.803255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-token\") pod \"e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") " pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:48.915505 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.915457    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:48.978439 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.978411    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"]
Sep 19 10:48:48.980977 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:48:48.980944    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89353aaf_f742_434b_9b75_f6fde2b8babc.slice/crio-395e466b9b1b07537c67d61ab9c25d6e01bcffcc55169d1c4c111eb52f5ba1e3 WatchSource:0}: Error finding container 395e466b9b1b07537c67d61ab9c25d6e01bcffcc55169d1c4c111eb52f5ba1e3: Status 404 returned error can't find the container with id 395e466b9b1b07537c67d61ab9c25d6e01bcffcc55169d1c4c111eb52f5ba1e3
Sep 19 10:48:48.984822 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:48.984808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:49.023773 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.023723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:49.113131 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.113097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:49.174692 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.174653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98" event={"ID":"89353aaf-f742-434b-9b75-f6fde2b8babc","Type":"ContainerStarted","Data":"395e466b9b1b07537c67d61ab9c25d6e01bcffcc55169d1c4c111eb52f5ba1e3"}
Sep 19 10:48:49.175852 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.175825    2580 generic.go:356] "Generic (PLEG): container finished" podID="2ead14de-e934-4fc9-9398-2e3db095701b" containerID="d425465b2004391742355fc0391b106004062444bac280ea2de2d7fde48f653d" exitCode=137
Sep 19 10:48:49.175997 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.175879    2580 scope.go:117] "RemoveContainer" containerID="d425465b2004391742355fc0391b106004062444bac280ea2de2d7fde48f653d"
Sep 19 10:48:49.175997 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.175876    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7980-7a335c"
Sep 19 10:48:49.177167 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.177011    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:49.185584 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.185562    2580 scope.go:117] "RemoveContainer" containerID="d425465b2004391742355fc0391b106004062444bac280ea2de2d7fde48f653d"
Sep 19 10:48:49.185952 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:48:49.185931    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d425465b2004391742355fc0391b106004062444bac280ea2de2d7fde48f653d\": container with ID starting with d425465b2004391742355fc0391b106004062444bac280ea2de2d7fde48f653d not found: ID does not exist" containerID="d425465b2004391742355fc0391b106004062444bac280ea2de2d7fde48f653d"
Sep 19 10:48:49.186040 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.185961    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d425465b2004391742355fc0391b106004062444bac280ea2de2d7fde48f653d"} err="failed to get container status \"d425465b2004391742355fc0391b106004062444bac280ea2de2d7fde48f653d\": rpc error: code = NotFound desc = could not find container \"d425465b2004391742355fc0391b106004062444bac280ea2de2d7fde48f653d\": container with ID starting with d425465b2004391742355fc0391b106004062444bac280ea2de2d7fde48f653d not found: ID does not exist"
Sep 19 10:48:49.186734 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:48:49.186705    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7980-7a335c\" is forbidden: User \"system:node:ip-10-0-129-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-220.ec2.internal' and this object" podUID="2ead14de-e934-4fc9-9398-2e3db095701b" pod="opendatahub/test-curl-7980-7a335c"
Sep 19 10:48:49.264903 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.264725    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"]
Sep 19 10:48:49.267394 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:48:49.267356    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e2ff36_97ab_440f_bcb3_cdfc65699add.slice/crio-7a91b4af09e5a384c8aa827d7d08546bca0cfe5d061b52621199bee2fcf7008b WatchSource:0}: Error finding container 7a91b4af09e5a384c8aa827d7d08546bca0cfe5d061b52621199bee2fcf7008b: Status 404 returned error can't find the container with id 7a91b4af09e5a384c8aa827d7d08546bca0cfe5d061b52621199bee2fcf7008b
Sep 19 10:48:49.479030 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.478989    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:49.790482 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.790445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:49.859642 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.859582    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ead14de-e934-4fc9-9398-2e3db095701b" path="/var/lib/kubelet/pods/2ead14de-e934-4fc9-9398-2e3db095701b/volumes"
Sep 19 10:48:49.916412 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:49.916364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:50.189227 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:50.189122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6" event={"ID":"89e2ff36-97ab-440f-bcb3-cdfc65699add","Type":"ContainerStarted","Data":"7a91b4af09e5a384c8aa827d7d08546bca0cfe5d061b52621199bee2fcf7008b"}
Sep 19 10:48:50.447001 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:50.446871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:50.505368 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:50.505315    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:50.699160 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:50.699049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:51.038335 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:51.038200    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:51.101594 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:51.101555    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:51.463693 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:51.463657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:52.430045 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:52.430010    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:52.477705 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:52.477676    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:48:52.478012 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:52.477738    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:48:52.478012 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:52.477786    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:48:53.200184 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:53.200140    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:53.205734 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:53.205701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6" event={"ID":"89e2ff36-97ab-440f-bcb3-cdfc65699add","Type":"ContainerStarted","Data":"4591fcd5d053d9dd76b58c6bc00314c11000f5651218bdb10373d11fb9ac9b9b"}
Sep 19 10:48:53.214474 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:53.214434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98" event={"ID":"89353aaf-f742-434b-9b75-f6fde2b8babc","Type":"ContainerStarted","Data":"5d2035d6261ed7a391edeedcae11a0576c9031a104285a886f887f504aa75b20"}
Sep 19 10:48:53.214632 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:53.214519    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:53.226737 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:53.226675    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6" podStartSLOduration=2.01916446 podStartE2EDuration="5.226653581s" totalImagesPullingTime="3.207489121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:48:53.225592398 +0000 UTC m=+777.884337559" watchObservedRunningTime="2026-09-19 10:48:53.226653581 +0000 UTC m=+777.885398744"
Sep 19 10:48:53.241826 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:53.241729    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98" podStartSLOduration=2.7493256280000002 podStartE2EDuration="6.241714389s" totalImagesPullingTime="3.492388761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:48:53.240158687 +0000 UTC m=+777.898903848" watchObservedRunningTime="2026-09-19 10:48:53.241714389 +0000 UTC m=+777.900459549"
Sep 19 10:48:53.985918 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:53.985873    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:53.990743 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:53.990718    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:54.221154 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:54.221118    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:54.221878 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:54.221856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:48:54.228048 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:54.228016    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:54.575786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:54.575734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:54.677266 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:54.677227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:55.148090 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:55.148054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:55.942039 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:55.941993    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:56.724974 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:56.724923    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:57.488315 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.488269    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:57.623026 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.622991    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"]
Sep 19 10:48:57.627980 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.627956    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:57.630101 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.630076    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c152d119-serving-cert\""
Sep 19 10:48:57.635795 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.635769    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"]
Sep 19 10:48:57.652240 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.652201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:57.676935 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.676900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-metrics-certs\") pod \"maas-api-e2e-ait-c152d119-674f68986f-mqddh\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:57.676935 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.676945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-maas-api-tls\") pod \"maas-api-e2e-ait-c152d119-674f68986f-mqddh\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:57.677148 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.676971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6rlt\" (UniqueName: \"kubernetes.io/projected/4b0c127e-fd87-40b8-842d-e55566aa7262-kube-api-access-k6rlt\") pod \"maas-api-e2e-ait-c152d119-674f68986f-mqddh\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:57.677148 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.677132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0c127e-fd87-40b8-842d-e55566aa7262-openshift-service-ca\") pod \"maas-api-e2e-ait-c152d119-674f68986f-mqddh\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:57.712403 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.712366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:57.778017 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.777934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0c127e-fd87-40b8-842d-e55566aa7262-openshift-service-ca\") pod \"maas-api-e2e-ait-c152d119-674f68986f-mqddh\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:57.778017 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.777974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-metrics-certs\") pod \"maas-api-e2e-ait-c152d119-674f68986f-mqddh\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:57.778017 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.778007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-maas-api-tls\") pod \"maas-api-e2e-ait-c152d119-674f68986f-mqddh\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:57.778499 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.778039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6rlt\" (UniqueName: \"kubernetes.io/projected/4b0c127e-fd87-40b8-842d-e55566aa7262-kube-api-access-k6rlt\") pod \"maas-api-e2e-ait-c152d119-674f68986f-mqddh\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:57.778499 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:48:57.778151    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-serving-cert: secret "maas-api-e2e-ait-c152d119-serving-cert" not found
Sep 19 10:48:57.778499 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:48:57.778215    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-maas-api-tls podName:4b0c127e-fd87-40b8-842d-e55566aa7262 nodeName:}" failed. No retries permitted until 2026-09-19 10:48:58.278198187 +0000 UTC m=+782.936943324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-maas-api-tls") pod "maas-api-e2e-ait-c152d119-674f68986f-mqddh" (UID: "4b0c127e-fd87-40b8-842d-e55566aa7262") : secret "maas-api-e2e-ait-c152d119-serving-cert" not found
Sep 19 10:48:57.778740 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.778722    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0c127e-fd87-40b8-842d-e55566aa7262-openshift-service-ca\") pod \"maas-api-e2e-ait-c152d119-674f68986f-mqddh\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:57.780661 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.780634    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-metrics-certs\") pod \"maas-api-e2e-ait-c152d119-674f68986f-mqddh\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:57.786645 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.786616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6rlt\" (UniqueName: \"kubernetes.io/projected/4b0c127e-fd87-40b8-842d-e55566aa7262-kube-api-access-k6rlt\") pod \"maas-api-e2e-ait-c152d119-674f68986f-mqddh\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:57.934101 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:57.934061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:58.034415 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:58.034291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:58.121129 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:58.121094    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:48:58.124075 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:58.124043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:58.186272 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:58.186222    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:58.282130 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:58.282099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-maas-api-tls\") pod \"maas-api-e2e-ait-c152d119-674f68986f-mqddh\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:58.284884 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:58.284807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-maas-api-tls\") pod \"maas-api-e2e-ait-c152d119-674f68986f-mqddh\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:58.332923 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:58.332887    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:58.436879 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:58.436843    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:58.541315 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:58.541224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:58.613990 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:58.613932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:58.727284 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:58.727244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:58.816304 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:58.816139    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"]
Sep 19 10:48:58.818547 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:58.818515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:58.819555 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:48:58.819531    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0c127e_fd87_40b8_842d_e55566aa7262.slice/crio-c5ea9625a855fba598c399a7ffa8d7674bc43181fdb4d847aab285ad3efb2a83 WatchSource:0}: Error finding container c5ea9625a855fba598c399a7ffa8d7674bc43181fdb4d847aab285ad3efb2a83: Status 404 returned error can't find the container with id c5ea9625a855fba598c399a7ffa8d7674bc43181fdb4d847aab285ad3efb2a83
Sep 19 10:48:59.029193 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:59.029150    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:48:59.226743 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:59.226718    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 10:48:59.247088 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:59.247056    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh" event={"ID":"4b0c127e-fd87-40b8-842d-e55566aa7262","Type":"ContainerStarted","Data":"1d45f62c50767eba6d1a61b642749034263b7603f3f476943478a5efd636d786"}
Sep 19 10:48:59.247249 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:59.247103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh" event={"ID":"4b0c127e-fd87-40b8-842d-e55566aa7262","Type":"ContainerStarted","Data":"c5ea9625a855fba598c399a7ffa8d7674bc43181fdb4d847aab285ad3efb2a83"}
Sep 19 10:48:59.247249 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:59.247134    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:48:59.268481 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:59.268417    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh" podStartSLOduration=1.9679772359999999 podStartE2EDuration="2.268399211s" totalImagesPullingTime="300.421975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:48:59.266951911 +0000 UTC m=+783.925697082" watchObservedRunningTime="2026-09-19 10:48:59.268399211 +0000 UTC m=+783.927144371"
Sep 19 10:48:59.835109 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:48:59.835067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:00.622190 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:00.622144    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:00.835835 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:00.835791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:01.082922 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:01.082879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:01.183243 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:01.183203    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:01.286706 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:01.286670    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:01.382264 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:01.382173    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:01.856884 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:01.856844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:02.891182 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:02.891145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:03.801320 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:03.801289    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:03.901615 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:03.901571    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:04.839442 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:04.839401    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:05.258376 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:05.258345    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:49:05.760817 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:05.760744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:05.861457 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:05.861408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:06.154958 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.154878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:06.324639 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.324604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:06.424463 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.424379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:06.428810 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.428782    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"]
Sep 19 10:49:06.432433 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.432403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.434606 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.434581    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-df95e5-openshift-default-dockercfg-s5f8l\""
Sep 19 10:49:06.446176 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.446149    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"]
Sep 19 10:49:06.506461 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.506423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:06.554428 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.554398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-data\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.554600 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.554448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-workload-socket\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.554600 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.554475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsxtt\" (UniqueName: \"kubernetes.io/projected/caa42699-c4cd-41a8-8825-61cb1b057d42-kube-api-access-zsxtt\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.554600 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.554523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-envoy\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.554600 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.554555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-workload-certs\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.554600 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.554574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-token\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.554834 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.554607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-credential-socket\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.554834 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.554632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-podinfo\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.554834 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.554654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa42699-c4cd-41a8-8825-61cb1b057d42-istiod-ca-cert\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.607743 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.607702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:06.655579 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.655542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-envoy\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.655783 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.655593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-workload-certs\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.655783 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.655623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-token\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.655783 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.655647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-credential-socket\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.655783 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.655672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-podinfo\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.655783 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.655698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa42699-c4cd-41a8-8825-61cb1b057d42-istiod-ca-cert\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.656040 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.655930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-data\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.656040 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.656020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-workload-socket\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.656140 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.656063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsxtt\" (UniqueName: \"kubernetes.io/projected/caa42699-c4cd-41a8-8825-61cb1b057d42-kube-api-access-zsxtt\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.656140 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.656107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-workload-certs\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.656245 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.656165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-credential-socket\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.656429 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.656401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-data\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.656429 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.656420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-workload-socket\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.656527 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.656431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa42699-c4cd-41a8-8825-61cb1b057d42-istiod-ca-cert\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.659072 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.659035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-podinfo\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.668421 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.663992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-envoy\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.674009 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.673979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-token\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.674169 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.674033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsxtt\" (UniqueName: \"kubernetes.io/projected/caa42699-c4cd-41a8-8825-61cb1b057d42-kube-api-access-zsxtt\") pod \"e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") " pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.743236 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.743199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:06.933057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:06.933016    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:07.031521 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:07.031496    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"]
Sep 19 10:49:07.033657 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:49:07.033630    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa42699_c4cd_41a8_8825_61cb1b057d42.slice/crio-a703826068f5a746199687d9892fb1fbf08041120ce642a8be0509bb4064ffac WatchSource:0}: Error finding container a703826068f5a746199687d9892fb1fbf08041120ce642a8be0509bb4064ffac: Status 404 returned error can't find the container with id a703826068f5a746199687d9892fb1fbf08041120ce642a8be0509bb4064ffac
Sep 19 10:49:07.036117 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:07.036078    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:07.036216 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:07.036173    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:07.036274 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:07.036214    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:07.078529 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:07.078488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:07.287735 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:07.287650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" event={"ID":"caa42699-c4cd-41a8-8825-61cb1b057d42","Type":"ContainerStarted","Data":"2d2aa346e190da13a7edbaa3f85f669b42ab61c71bf4c5cc9f6918af5d07ba88"}
Sep 19 10:49:07.287735 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:07.287711    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" event={"ID":"caa42699-c4cd-41a8-8825-61cb1b057d42","Type":"ContainerStarted","Data":"a703826068f5a746199687d9892fb1fbf08041120ce642a8be0509bb4064ffac"}
Sep 19 10:49:07.310202 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:07.310142    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" podStartSLOduration=1.310119474 podStartE2EDuration="1.310119474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:49:07.307597955 +0000 UTC m=+791.966343116" watchObservedRunningTime="2026-09-19 10:49:07.310119474 +0000 UTC m=+791.968864634"
Sep 19 10:49:07.739644 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:07.739606    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:07.743375 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:07.743353    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:08.711701 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:08.711661    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:08.748857 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:08.748828    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:08.811723 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:08.811691    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:09.172899 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:09.172806    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:09.274810 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:09.274747    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:09.296938 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:09.296907    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:09.297838 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:09.297820    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:09.641173 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:09.641136    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:10.740943 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:10.740907    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:11.432246 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:11.432211    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:12.241386 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:12.241347    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:12.432785 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:12.432733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:12.527426 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:12.527334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:12.626161 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:12.626122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:12.745671 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:12.745633    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:13.097733 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.097697    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"]
Sep 19 10:49:13.101740 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.101715    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.104251 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.104227    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-df95e5-serving-cert\""
Sep 19 10:49:13.114310 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.114289    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"]
Sep 19 10:49:13.134805 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.134778    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:13.219326 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.219290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d74ck\" (UniqueName: \"kubernetes.io/projected/2017702a-00f7-487f-babd-0057c8fdcbed-kube-api-access-d74ck\") pod \"maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.219522 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.219335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-metrics-certs\") pod \"maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.219522 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.219392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2017702a-00f7-487f-babd-0057c8fdcbed-openshift-service-ca\") pod \"maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.219522 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.219500    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-maas-api-tls\") pod \"maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.232116 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.232080    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:13.320022 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.319988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d74ck\" (UniqueName: \"kubernetes.io/projected/2017702a-00f7-487f-babd-0057c8fdcbed-kube-api-access-d74ck\") pod \"maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.320494 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.320035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-metrics-certs\") pod \"maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.320494 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.320084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2017702a-00f7-487f-babd-0057c8fdcbed-openshift-service-ca\") pod \"maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.320494 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.320163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-maas-api-tls\") pod \"maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.320494 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:49:13.320272    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-serving-cert: secret "maas-api-e2e-inf-b-df95e5-serving-cert" not found
Sep 19 10:49:13.320494 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:49:13.320376    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-maas-api-tls podName:2017702a-00f7-487f-babd-0057c8fdcbed nodeName:}" failed. No retries permitted until 2026-09-19 10:49:13.820353677 +0000 UTC m=+798.479098834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-maas-api-tls") pod "maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m" (UID: "2017702a-00f7-487f-babd-0057c8fdcbed") : secret "maas-api-e2e-inf-b-df95e5-serving-cert" not found
Sep 19 10:49:13.320826 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.320807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2017702a-00f7-487f-babd-0057c8fdcbed-openshift-service-ca\") pod \"maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.322697 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.322675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-metrics-certs\") pod \"maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.331015 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.330990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d74ck\" (UniqueName: \"kubernetes.io/projected/2017702a-00f7-487f-babd-0057c8fdcbed-kube-api-access-d74ck\") pod \"maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.635985 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.635946    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:13.741082 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.741051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:13.825189 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.825149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-maas-api-tls\") pod \"maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.827880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.827858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-maas-api-tls\") pod \"maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:13.855192 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:13.855138    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:14.015378 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:14.015336    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:14.127732 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:14.127692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:14.314509 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:14.314486    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"]
Sep 19 10:49:14.316742 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:49:14.316714    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2017702a_00f7_487f_babd_0057c8fdcbed.slice/crio-7529ff82366a591c1865d51418325c30f387109eb57ad50b0523f9780a2a3f2b WatchSource:0}: Error finding container 7529ff82366a591c1865d51418325c30f387109eb57ad50b0523f9780a2a3f2b: Status 404 returned error can't find the container with id 7529ff82366a591c1865d51418325c30f387109eb57ad50b0523f9780a2a3f2b
Sep 19 10:49:15.022819 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:15.022775    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:15.327749 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:15.327640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m" event={"ID":"2017702a-00f7-487f-babd-0057c8fdcbed","Type":"ContainerStarted","Data":"25793f1db31541227570da12454f357f0d50e749642167ae7f39ee876fc137f0"}
Sep 19 10:49:15.327749 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:15.327700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m" event={"ID":"2017702a-00f7-487f-babd-0057c8fdcbed","Type":"ContainerStarted","Data":"7529ff82366a591c1865d51418325c30f387109eb57ad50b0523f9780a2a3f2b"}
Sep 19 10:49:15.327749 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:15.327740    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:15.346362 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:15.346308    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m" podStartSLOduration=2.046934764 podStartE2EDuration="2.346289424s" totalImagesPullingTime="299.35466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:49:15.344888225 +0000 UTC m=+800.003633388" watchObservedRunningTime="2026-09-19 10:49:15.346289424 +0000 UTC m=+800.005034584"
Sep 19 10:49:15.747877 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:15.747836    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:16.423258 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:16.423219    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:17.815310 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:17.815262    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:49:18.345006 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:18.344976    2580 generic.go:356] "Generic (PLEG): container finished" podID="89e2ff36-97ab-440f-bcb3-cdfc65699add" containerID="4591fcd5d053d9dd76b58c6bc00314c11000f5651218bdb10373d11fb9ac9b9b" exitCode=0
Sep 19 10:49:18.345178 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:18.345052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6" event={"ID":"89e2ff36-97ab-440f-bcb3-cdfc65699add","Type":"ContainerDied","Data":"4591fcd5d053d9dd76b58c6bc00314c11000f5651218bdb10373d11fb9ac9b9b"}
Sep 19 10:49:18.345502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:18.345484    2580 scope.go:117] "RemoveContainer" containerID="4591fcd5d053d9dd76b58c6bc00314c11000f5651218bdb10373d11fb9ac9b9b"
Sep 19 10:49:18.346626 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:18.346596    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:18.346698 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:18.346667    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:18.346736 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:18.346711    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:18.839576 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:18.839535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:18.940431 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:18.940398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:19.040231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:19.040194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:19.351680 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:19.351641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6" event={"ID":"89e2ff36-97ab-440f-bcb3-cdfc65699add","Type":"ContainerStarted","Data":"8255c4087d260f8c13eded573719105ac105c0057e2cb65c820e3845b1bf01ca"}
Sep 19 10:49:19.986096 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:19.986060    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:49:19.987069 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:19.987036    2580 patch_prober.go:28] interesting pod/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 19 10:49:19.987190 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:19.987116    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6" podUID="89e2ff36-97ab-440f-bcb3-cdfc65699add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 19 10:49:20.243477 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.243374    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"]
Sep 19 10:49:20.243643 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.243619    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh" podUID="4b0c127e-fd87-40b8-842d-e55566aa7262" containerName="maas-api" containerID="cri-o://1d45f62c50767eba6d1a61b642749034263b7603f3f476943478a5efd636d786" gracePeriod=30
Sep 19 10:49:20.253022 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.252984    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh" podUID="4b0c127e-fd87-40b8-842d-e55566aa7262" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.37:8443/health\": dial tcp 10.132.0.37:8443: connect: connection refused"
Sep 19 10:49:20.267002 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.266961    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:20.360368 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.360337    2580 generic.go:356] "Generic (PLEG): container finished" podID="4b0c127e-fd87-40b8-842d-e55566aa7262" containerID="1d45f62c50767eba6d1a61b642749034263b7603f3f476943478a5efd636d786" exitCode=0
Sep 19 10:49:20.360616 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.360415    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh" event={"ID":"4b0c127e-fd87-40b8-842d-e55566aa7262","Type":"ContainerDied","Data":"1d45f62c50767eba6d1a61b642749034263b7603f3f476943478a5efd636d786"}
Sep 19 10:49:20.364311 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.364257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:20.472355 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.472244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:20.523470 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.523445    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:49:20.550207 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.550178    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:20.588782 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.588067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0c127e-fd87-40b8-842d-e55566aa7262-openshift-service-ca\") pod \"4b0c127e-fd87-40b8-842d-e55566aa7262\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") "
Sep 19 10:49:20.588782 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.588135    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-maas-api-tls\") pod \"4b0c127e-fd87-40b8-842d-e55566aa7262\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") "
Sep 19 10:49:20.588782 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.588215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6rlt\" (UniqueName: \"kubernetes.io/projected/4b0c127e-fd87-40b8-842d-e55566aa7262-kube-api-access-k6rlt\") pod \"4b0c127e-fd87-40b8-842d-e55566aa7262\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") "
Sep 19 10:49:20.588782 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.588251    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-metrics-certs\") pod \"4b0c127e-fd87-40b8-842d-e55566aa7262\" (UID: \"4b0c127e-fd87-40b8-842d-e55566aa7262\") "
Sep 19 10:49:20.589779 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.589591    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0c127e-fd87-40b8-842d-e55566aa7262-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b0c127e-fd87-40b8-842d-e55566aa7262" (UID: "4b0c127e-fd87-40b8-842d-e55566aa7262"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:49:20.594152 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.593140    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4b0c127e-fd87-40b8-842d-e55566aa7262" (UID: "4b0c127e-fd87-40b8-842d-e55566aa7262"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:49:20.594152 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.593521    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0c127e-fd87-40b8-842d-e55566aa7262-kube-api-access-k6rlt" (OuterVolumeSpecName: "kube-api-access-k6rlt") pod "4b0c127e-fd87-40b8-842d-e55566aa7262" (UID: "4b0c127e-fd87-40b8-842d-e55566aa7262"). InnerVolumeSpecName "kube-api-access-k6rlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:49:20.595390 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.595358    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b0c127e-fd87-40b8-842d-e55566aa7262" (UID: "4b0c127e-fd87-40b8-842d-e55566aa7262"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:49:20.669192 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.669147    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"]
Sep 19 10:49:20.689473 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.689419    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0c127e-fd87-40b8-842d-e55566aa7262-openshift-service-ca\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:20.689473 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.689466    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-maas-api-tls\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:20.689473 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.689482    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6rlt\" (UniqueName: \"kubernetes.io/projected/4b0c127e-fd87-40b8-842d-e55566aa7262-kube-api-access-k6rlt\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:20.689830 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.689494    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0c127e-fd87-40b8-842d-e55566aa7262-metrics-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:20.701328 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.701293    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:20.919596 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.919435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:20.986308 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.985858    2580 patch_prober.go:28] interesting pod/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 19 10:49:20.986308 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:20.985919    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6" podUID="89e2ff36-97ab-440f-bcb3-cdfc65699add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 19 10:49:21.083991 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.083815    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:21.155344 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.155305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:21.299595 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.299546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:21.340581 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.340550    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:49:21.375781 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.372449    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6" podUID="89e2ff36-97ab-440f-bcb3-cdfc65699add" containerName="istio-proxy" containerID="cri-o://8255c4087d260f8c13eded573719105ac105c0057e2cb65c820e3845b1bf01ca" gracePeriod=30
Sep 19 10:49:21.375781 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.372887    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"
Sep 19 10:49:21.378432 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.378374    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh" event={"ID":"4b0c127e-fd87-40b8-842d-e55566aa7262","Type":"ContainerDied","Data":"c5ea9625a855fba598c399a7ffa8d7674bc43181fdb4d847aab285ad3efb2a83"}
Sep 19 10:49:21.378546 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.378526    2580 scope.go:117] "RemoveContainer" containerID="1d45f62c50767eba6d1a61b642749034263b7603f3f476943478a5efd636d786"
Sep 19 10:49:21.438415 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.438380    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"]
Sep 19 10:49:21.441970 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.441945    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c152d119-674f68986f-mqddh"]
Sep 19 10:49:21.586835 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.586001    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"]
Sep 19 10:49:21.586835 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.586582    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0c127e-fd87-40b8-842d-e55566aa7262" containerName="maas-api"
Sep 19 10:49:21.586835 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.586602    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0c127e-fd87-40b8-842d-e55566aa7262" containerName="maas-api"
Sep 19 10:49:21.586835 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.586701    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0c127e-fd87-40b8-842d-e55566aa7262" containerName="maas-api"
Sep 19 10:49:21.606776 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.594461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.606776 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.601557    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"]
Sep 19 10:49:21.606776 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.601832    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-18e8cec5-gw-openshift-default-dockercfg-59r7q\""
Sep 19 10:49:21.633390 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.633172    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:21.708784 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.708223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-token\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.708784 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.708282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-data\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.708784 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.708328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istiod-ca-cert\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.708784 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.708363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzprw\" (UniqueName: \"kubernetes.io/projected/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-kube-api-access-jzprw\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.708784 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.708391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-workload-socket\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.708784 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.708414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-workload-certs\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.708784 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.708456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-credential-socket\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.708784 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.708486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-envoy\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.708784 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.708509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-podinfo\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.747633 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.747549    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:21.809953 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.809904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-token\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.810167 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.809970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-data\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.810167 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.810020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istiod-ca-cert\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.810167 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.810059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzprw\" (UniqueName: \"kubernetes.io/projected/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-kube-api-access-jzprw\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.810167 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.810089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-workload-socket\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.810167 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.810117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-workload-certs\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.810167 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.810161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-credential-socket\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.810501 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.810192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-envoy\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.810501 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.810215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-podinfo\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.811511 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.811477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istiod-ca-cert\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.811648 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.811559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-workload-certs\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.811817 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.811795    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-workload-socket\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.811938 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.811914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-credential-socket\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.812708 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.812686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-data\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.814223 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.814146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-podinfo\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.815803 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.814697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-envoy\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.819186 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.819160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-token\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.819491 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.819472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzprw\" (UniqueName: \"kubernetes.io/projected/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-kube-api-access-jzprw\") pod \"e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") " pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:21.857023 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.856928    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b0c127e-fd87-40b8-842d-e55566aa7262" path="/var/lib/kubelet/pods/4b0c127e-fd87-40b8-842d-e55566aa7262/volumes"
Sep 19 10:49:21.926940 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.926895    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:21.937904 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:21.937875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:22.263873 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:22.262213    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"]
Sep 19 10:49:22.284616 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:22.283687    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:22.284616 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:22.283806    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:22.284616 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:22.283856    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:22.384159 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:22.384119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" event={"ID":"a98a631a-d59d-4be6-8bf9-dbad0d605fd9","Type":"ContainerStarted","Data":"b053032dc46bfe43ddcd17cd6af8c494cd1d9f6f454aec6a21d874550cb9bebe"}
Sep 19 10:49:22.384369 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:22.384184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" event={"ID":"a98a631a-d59d-4be6-8bf9-dbad0d605fd9","Type":"ContainerStarted","Data":"0e01bff58af337c01b52f52bb17e14a9ed42aac8a95a7ba63158c9ddebc02a11"}
Sep 19 10:49:22.405032 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:22.404962    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" podStartSLOduration=1.404939215 podStartE2EDuration="1.404939215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:49:22.401876423 +0000 UTC m=+807.060621584" watchObservedRunningTime="2026-09-19 10:49:22.404939215 +0000 UTC m=+807.063684377"
Sep 19 10:49:22.630588 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:22.630459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:22.938036 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:22.938001    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:23.429048 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:23.428950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:23.846979 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:23.846940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:23.943518 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:23.943484    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:23.945467 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:23.945433    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:24.221892 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:24.221855    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:49:24.383960 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:24.383920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:24.401868 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:24.401824    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:24.410388 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:24.410350    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:25.139536 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:25.139498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:25.906553 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:25.906515    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"]
Sep 19 10:49:25.918311 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:25.918259    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:25.921561 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:25.920854    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-18e8cec5-serving-cert\""
Sep 19 10:49:25.923592 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:25.923555    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"]
Sep 19 10:49:25.938217 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:25.938170    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:26.034386 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.034329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:26.055453 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.055407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-maas-api-tls\") pod \"maas-api-e2e-migrate-18e8cec5-f5858f664-klq68\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:26.055618 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.055457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/130e6f58-a3b9-4601-a39d-9628b9837801-openshift-service-ca\") pod \"maas-api-e2e-migrate-18e8cec5-f5858f664-klq68\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:26.055618 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.055578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxs4r\" (UniqueName: \"kubernetes.io/projected/130e6f58-a3b9-4601-a39d-9628b9837801-kube-api-access-xxs4r\") pod \"maas-api-e2e-migrate-18e8cec5-f5858f664-klq68\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:26.055698 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.055624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-metrics-certs\") pod \"maas-api-e2e-migrate-18e8cec5-f5858f664-klq68\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:26.156911 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.156808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-metrics-certs\") pod \"maas-api-e2e-migrate-18e8cec5-f5858f664-klq68\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:26.156911 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.156887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-maas-api-tls\") pod \"maas-api-e2e-migrate-18e8cec5-f5858f664-klq68\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:26.157436 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.156914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/130e6f58-a3b9-4601-a39d-9628b9837801-openshift-service-ca\") pod \"maas-api-e2e-migrate-18e8cec5-f5858f664-klq68\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:26.157436 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.156977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxs4r\" (UniqueName: \"kubernetes.io/projected/130e6f58-a3b9-4601-a39d-9628b9837801-kube-api-access-xxs4r\") pod \"maas-api-e2e-migrate-18e8cec5-f5858f664-klq68\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:26.157436 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:49:26.157025    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-serving-cert: secret "maas-api-e2e-migrate-18e8cec5-serving-cert" not found
Sep 19 10:49:26.157436 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:49:26.157139    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-maas-api-tls podName:130e6f58-a3b9-4601-a39d-9628b9837801 nodeName:}" failed. No retries permitted until 2026-09-19 10:49:26.657114283 +0000 UTC m=+811.315859436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-maas-api-tls") pod "maas-api-e2e-migrate-18e8cec5-f5858f664-klq68" (UID: "130e6f58-a3b9-4601-a39d-9628b9837801") : secret "maas-api-e2e-migrate-18e8cec5-serving-cert" not found
Sep 19 10:49:26.157920 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.157887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/130e6f58-a3b9-4601-a39d-9628b9837801-openshift-service-ca\") pod \"maas-api-e2e-migrate-18e8cec5-f5858f664-klq68\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:26.160076 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.160047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-metrics-certs\") pod \"maas-api-e2e-migrate-18e8cec5-f5858f664-klq68\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:26.166865 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.166826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxs4r\" (UniqueName: \"kubernetes.io/projected/130e6f58-a3b9-4601-a39d-9628b9837801-kube-api-access-xxs4r\") pod \"maas-api-e2e-migrate-18e8cec5-f5858f664-klq68\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:26.283370 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.283320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:26.456573 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.456538    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:26.555886 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.555850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:26.664358 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.664314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-maas-api-tls\") pod \"maas-api-e2e-migrate-18e8cec5-f5858f664-klq68\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:26.667487 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.667457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-maas-api-tls\") pod \"maas-api-e2e-migrate-18e8cec5-f5858f664-klq68\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:26.822043 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.821948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:26.834855 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:26.834818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:27.111458 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:27.111428    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"]
Sep 19 10:49:27.113413 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:49:27.113388    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130e6f58_a3b9_4601_a39d_9628b9837801.slice/crio-7f602c5e24d175c570949c2b232df5d787153060bb9f42bbfbfa68107fdf01c2 WatchSource:0}: Error finding container 7f602c5e24d175c570949c2b232df5d787153060bb9f42bbfbfa68107fdf01c2: Status 404 returned error can't find the container with id 7f602c5e24d175c570949c2b232df5d787153060bb9f42bbfbfa68107fdf01c2
Sep 19 10:49:27.436230 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:27.436190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68" event={"ID":"130e6f58-a3b9-4601-a39d-9628b9837801","Type":"ContainerStarted","Data":"7f602c5e24d175c570949c2b232df5d787153060bb9f42bbfbfa68107fdf01c2"}
Sep 19 10:49:27.556418 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:27.556372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:28.202994 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:28.202964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:28.444431 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:28.444399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68" event={"ID":"130e6f58-a3b9-4601-a39d-9628b9837801","Type":"ContainerStarted","Data":"edd35e22a3df70fa2bab0cf3df3e884ba4e062ee8e6ea723bf76e5547a05a425"}
Sep 19 10:49:28.444857 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:28.444460    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:28.461948 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:28.461852    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68" podStartSLOduration=2.998620999 podStartE2EDuration="3.461835191s" totalImagesPullingTime="463.214192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:49:28.459948945 +0000 UTC m=+813.118694104" watchObservedRunningTime="2026-09-19 10:49:28.461835191 +0000 UTC m=+813.120580351"
Sep 19 10:49:28.912067 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:28.911977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:29.015475 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:29.015448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:29.538335 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:29.538293    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:30.243638 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:30.243603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:30.457143 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:30.457113    2580 generic.go:356] "Generic (PLEG): container finished" podID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerID="2d2aa346e190da13a7edbaa3f85f669b42ab61c71bf4c5cc9f6918af5d07ba88" exitCode=0
Sep 19 10:49:30.457333 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:30.457189    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" event={"ID":"caa42699-c4cd-41a8-8825-61cb1b057d42","Type":"ContainerDied","Data":"2d2aa346e190da13a7edbaa3f85f669b42ab61c71bf4c5cc9f6918af5d07ba88"}
Sep 19 10:49:30.457591 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:30.457577    2580 scope.go:117] "RemoveContainer" containerID="2d2aa346e190da13a7edbaa3f85f669b42ab61c71bf4c5cc9f6918af5d07ba88"
Sep 19 10:49:30.458988 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:30.458955    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:30.459123 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:30.459042    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:30.459123 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:30.459096    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:30.866703 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:30.866610    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:31.468647 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.468596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" event={"ID":"caa42699-c4cd-41a8-8825-61cb1b057d42","Type":"ContainerStarted","Data":"f4a02b0be6984ccc1bb58c5b268bfde35355fca4c40603dfac6f57f3cbb7442b"}
Sep 19 10:49:31.673783 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.671971    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:49:31.743409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.743312    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:31.743700 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.743676    2580 patch_prober.go:28] interesting pod/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 19 10:49:31.743833 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.743733    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 19 10:49:31.813632 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.813600    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-credential-socket\") pod \"89e2ff36-97ab-440f-bcb3-cdfc65699add\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") "
Sep 19 10:49:31.813632 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.813638    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-workload-certs\") pod \"89e2ff36-97ab-440f-bcb3-cdfc65699add\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") "
Sep 19 10:49:31.813926 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.813679    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-envoy\") pod \"89e2ff36-97ab-440f-bcb3-cdfc65699add\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") "
Sep 19 10:49:31.813926 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.813713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e2ff36-97ab-440f-bcb3-cdfc65699add-istiod-ca-cert\") pod \"89e2ff36-97ab-440f-bcb3-cdfc65699add\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") "
Sep 19 10:49:31.813926 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.813808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-token\") pod \"89e2ff36-97ab-440f-bcb3-cdfc65699add\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") "
Sep 19 10:49:31.813926 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.813863    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx4wg\" (UniqueName: \"kubernetes.io/projected/89e2ff36-97ab-440f-bcb3-cdfc65699add-kube-api-access-bx4wg\") pod \"89e2ff36-97ab-440f-bcb3-cdfc65699add\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") "
Sep 19 10:49:31.813926 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.813896    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-data\") pod \"89e2ff36-97ab-440f-bcb3-cdfc65699add\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") "
Sep 19 10:49:31.814183 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.813934    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-podinfo\") pod \"89e2ff36-97ab-440f-bcb3-cdfc65699add\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") "
Sep 19 10:49:31.814183 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.813936    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "89e2ff36-97ab-440f-bcb3-cdfc65699add" (UID: "89e2ff36-97ab-440f-bcb3-cdfc65699add"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:49:31.814183 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.813967    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "89e2ff36-97ab-440f-bcb3-cdfc65699add" (UID: "89e2ff36-97ab-440f-bcb3-cdfc65699add"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:49:31.814183 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.813975    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-workload-socket\") pod \"89e2ff36-97ab-440f-bcb3-cdfc65699add\" (UID: \"89e2ff36-97ab-440f-bcb3-cdfc65699add\") "
Sep 19 10:49:31.814183 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.814090    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e2ff36-97ab-440f-bcb3-cdfc65699add-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "89e2ff36-97ab-440f-bcb3-cdfc65699add" (UID: "89e2ff36-97ab-440f-bcb3-cdfc65699add"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:49:31.814183 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.814120    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "89e2ff36-97ab-440f-bcb3-cdfc65699add" (UID: "89e2ff36-97ab-440f-bcb3-cdfc65699add"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:49:31.814495 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.814208    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-data" (OuterVolumeSpecName: "istio-data") pod "89e2ff36-97ab-440f-bcb3-cdfc65699add" (UID: "89e2ff36-97ab-440f-bcb3-cdfc65699add"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:49:31.814495 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.814319    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-data\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:31.814495 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.814336    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-workload-socket\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:31.814495 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.814349    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-credential-socket\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:31.814495 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.814363    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-workload-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:31.814495 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.814376    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e2ff36-97ab-440f-bcb3-cdfc65699add-istiod-ca-cert\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:31.816134 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.816106    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "89e2ff36-97ab-440f-bcb3-cdfc65699add" (UID: "89e2ff36-97ab-440f-bcb3-cdfc65699add"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:49:31.816299 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.816275    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "89e2ff36-97ab-440f-bcb3-cdfc65699add" (UID: "89e2ff36-97ab-440f-bcb3-cdfc65699add"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 10:49:31.816353 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.816303    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e2ff36-97ab-440f-bcb3-cdfc65699add-kube-api-access-bx4wg" (OuterVolumeSpecName: "kube-api-access-bx4wg") pod "89e2ff36-97ab-440f-bcb3-cdfc65699add" (UID: "89e2ff36-97ab-440f-bcb3-cdfc65699add"). InnerVolumeSpecName "kube-api-access-bx4wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:49:31.816391 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.816365    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-token" (OuterVolumeSpecName: "istio-token") pod "89e2ff36-97ab-440f-bcb3-cdfc65699add" (UID: "89e2ff36-97ab-440f-bcb3-cdfc65699add"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:49:31.915424 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.915383    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-podinfo\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:31.915424 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.915413    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-envoy\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:31.915424 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.915430    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e2ff36-97ab-440f-bcb3-cdfc65699add-istio-token\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:31.915921 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:31.915440    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx4wg\" (UniqueName: \"kubernetes.io/projected/89e2ff36-97ab-440f-bcb3-cdfc65699add-kube-api-access-bx4wg\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:32.479959 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.479913    2580 generic.go:356] "Generic (PLEG): container finished" podID="89e2ff36-97ab-440f-bcb3-cdfc65699add" containerID="8255c4087d260f8c13eded573719105ac105c0057e2cb65c820e3845b1bf01ca" exitCode=0
Sep 19 10:49:32.480728 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.480698    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"
Sep 19 10:49:32.481328 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.481302    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6" event={"ID":"89e2ff36-97ab-440f-bcb3-cdfc65699add","Type":"ContainerDied","Data":"8255c4087d260f8c13eded573719105ac105c0057e2cb65c820e3845b1bf01ca"}
Sep 19 10:49:32.481434 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.481356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6" event={"ID":"89e2ff36-97ab-440f-bcb3-cdfc65699add","Type":"ContainerDied","Data":"7a91b4af09e5a384c8aa827d7d08546bca0cfe5d061b52621199bee2fcf7008b"}
Sep 19 10:49:32.481434 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.481385    2580 scope.go:117] "RemoveContainer" containerID="8255c4087d260f8c13eded573719105ac105c0057e2cb65c820e3845b1bf01ca"
Sep 19 10:49:32.494457 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.494435    2580 scope.go:117] "RemoveContainer" containerID="4591fcd5d053d9dd76b58c6bc00314c11000f5651218bdb10373d11fb9ac9b9b"
Sep 19 10:49:32.497920 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.497896    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"]
Sep 19 10:49:32.501205 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.501183    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c152d119-openshift-default-5cfb749d64-wr7x6"]
Sep 19 10:49:32.503617 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.503459    2580 scope.go:117] "RemoveContainer" containerID="8255c4087d260f8c13eded573719105ac105c0057e2cb65c820e3845b1bf01ca"
Sep 19 10:49:32.503766 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:49:32.503737    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8255c4087d260f8c13eded573719105ac105c0057e2cb65c820e3845b1bf01ca\": container with ID starting with 8255c4087d260f8c13eded573719105ac105c0057e2cb65c820e3845b1bf01ca not found: ID does not exist" containerID="8255c4087d260f8c13eded573719105ac105c0057e2cb65c820e3845b1bf01ca"
Sep 19 10:49:32.503818 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.503780    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8255c4087d260f8c13eded573719105ac105c0057e2cb65c820e3845b1bf01ca"} err="failed to get container status \"8255c4087d260f8c13eded573719105ac105c0057e2cb65c820e3845b1bf01ca\": rpc error: code = NotFound desc = could not find container \"8255c4087d260f8c13eded573719105ac105c0057e2cb65c820e3845b1bf01ca\": container with ID starting with 8255c4087d260f8c13eded573719105ac105c0057e2cb65c820e3845b1bf01ca not found: ID does not exist"
Sep 19 10:49:32.503818 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.503801    2580 scope.go:117] "RemoveContainer" containerID="4591fcd5d053d9dd76b58c6bc00314c11000f5651218bdb10373d11fb9ac9b9b"
Sep 19 10:49:32.504102 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:49:32.504070    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4591fcd5d053d9dd76b58c6bc00314c11000f5651218bdb10373d11fb9ac9b9b\": container with ID starting with 4591fcd5d053d9dd76b58c6bc00314c11000f5651218bdb10373d11fb9ac9b9b not found: ID does not exist" containerID="4591fcd5d053d9dd76b58c6bc00314c11000f5651218bdb10373d11fb9ac9b9b"
Sep 19 10:49:32.504148 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.504110    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4591fcd5d053d9dd76b58c6bc00314c11000f5651218bdb10373d11fb9ac9b9b"} err="failed to get container status \"4591fcd5d053d9dd76b58c6bc00314c11000f5651218bdb10373d11fb9ac9b9b\": rpc error: code = NotFound desc = could not find container \"4591fcd5d053d9dd76b58c6bc00314c11000f5651218bdb10373d11fb9ac9b9b\": container with ID starting with 4591fcd5d053d9dd76b58c6bc00314c11000f5651218bdb10373d11fb9ac9b9b not found: ID does not exist"
Sep 19 10:49:32.743785 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.743673    2580 patch_prober.go:28] interesting pod/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 19 10:49:32.743785 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:32.743728    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 19 10:49:33.744179 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:33.744144    2580 patch_prober.go:28] interesting pod/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 19 10:49:33.744594 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:33.744202    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 19 10:49:33.857405 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:33.857375    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89e2ff36-97ab-440f-bcb3-cdfc65699add" path="/var/lib/kubelet/pods/89e2ff36-97ab-440f-bcb3-cdfc65699add/volumes"
Sep 19 10:49:34.102876 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:34.102771    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:34.455813 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:34.455781    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:34.745009 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:34.744921    2580 patch_prober.go:28] interesting pod/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 19 10:49:34.745009 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:34.744996    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 19 10:49:35.744602 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:35.744561    2580 patch_prober.go:28] interesting pod/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 19 10:49:35.744832 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:35.744617    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 19 10:49:36.800339 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:36.800303    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:36.800858 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:36.800679    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:36.801700 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:36.801677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:49:40.442983 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:40.442937    2580 patch_prober.go:28] interesting pod/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 19 10:49:40.443361 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:40.443014    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 19 10:49:42.542631 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.542599    2580 generic.go:356] "Generic (PLEG): container finished" podID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerID="b053032dc46bfe43ddcd17cd6af8c494cd1d9f6f454aec6a21d874550cb9bebe" exitCode=0
Sep 19 10:49:42.543075 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.542677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" event={"ID":"a98a631a-d59d-4be6-8bf9-dbad0d605fd9","Type":"ContainerDied","Data":"b053032dc46bfe43ddcd17cd6af8c494cd1d9f6f454aec6a21d874550cb9bebe"}
Sep 19 10:49:42.543150 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.543134    2580 scope.go:117] "RemoveContainer" containerID="b053032dc46bfe43ddcd17cd6af8c494cd1d9f6f454aec6a21d874550cb9bebe"
Sep 19 10:49:42.544318 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.544288    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:42.544384 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.544367    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:42.544420 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.544398    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:42.696495 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.696465    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"]
Sep 19 10:49:42.696891 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.696876    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89e2ff36-97ab-440f-bcb3-cdfc65699add" containerName="istio-proxy"
Sep 19 10:49:42.696947 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.696893    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89e2ff36-97ab-440f-bcb3-cdfc65699add" containerName="istio-proxy"
Sep 19 10:49:42.696947 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.696902    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89e2ff36-97ab-440f-bcb3-cdfc65699add" containerName="istio-proxy"
Sep 19 10:49:42.696947 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.696908    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89e2ff36-97ab-440f-bcb3-cdfc65699add" containerName="istio-proxy"
Sep 19 10:49:42.697061 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.696966    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="89e2ff36-97ab-440f-bcb3-cdfc65699add" containerName="istio-proxy"
Sep 19 10:49:42.700064 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.700043    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.703015 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.702820    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5116ec\"/\"kube-root-ca.crt\""
Sep 19 10:49:42.703015 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.702850    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5116ec\"/\"default-dockercfg-ghbth\""
Sep 19 10:49:42.703193 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.703147    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5116ec\"/\"openshift-service-ca.crt\""
Sep 19 10:49:42.703193 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.703148    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5116ec\"/\"simulator-w4-5116ec-kserve-self-signed-certs\""
Sep 19 10:49:42.711039 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.711009    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"]
Sep 19 10:49:42.725407 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.725375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-home\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.725601 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.725419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-tmp-dir\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.725601 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.725457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7psxl\" (UniqueName: \"kubernetes.io/projected/c47630c3-6941-4929-8c74-a79aa06e92a8-kube-api-access-7psxl\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.725601 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.725486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-dshm\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.725601 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.725519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-model-cache\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.725601 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.725574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47630c3-6941-4929-8c74-a79aa06e92a8-tls-certs\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.826171 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.826074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-tmp-dir\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.826171 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.826114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7psxl\" (UniqueName: \"kubernetes.io/projected/c47630c3-6941-4929-8c74-a79aa06e92a8-kube-api-access-7psxl\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.826171 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.826141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-dshm\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.826171 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.826165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-model-cache\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.826968 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.826191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47630c3-6941-4929-8c74-a79aa06e92a8-tls-certs\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.826968 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.826254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-home\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.826968 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.826506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-tmp-dir\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.826968 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.826552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-model-cache\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.826968 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.826620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-home\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.828913 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.828886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-dshm\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.829095 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.829073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47630c3-6941-4929-8c74-a79aa06e92a8-tls-certs\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:42.834142 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:42.834114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7psxl\" (UniqueName: \"kubernetes.io/projected/c47630c3-6941-4929-8c74-a79aa06e92a8-kube-api-access-7psxl\") pod \"simulator-w4-5116ec-kserve-7c6597457d-vpz59\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") " pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:43.012366 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:43.012319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:43.206253 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:43.206165    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:43.219828 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:43.219790    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:49:43.298242 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:43.298112    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"]
Sep 19 10:49:43.301173 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:49:43.301134    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc47630c3_6941_4929_8c74_a79aa06e92a8.slice/crio-1539ba59740558c31ad41c18f0eb69dabeae48e53568467dcf3f162562008c1b WatchSource:0}: Error finding container 1539ba59740558c31ad41c18f0eb69dabeae48e53568467dcf3f162562008c1b: Status 404 returned error can't find the container with id 1539ba59740558c31ad41c18f0eb69dabeae48e53568467dcf3f162562008c1b
Sep 19 10:49:43.549233 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:43.549134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" event={"ID":"a98a631a-d59d-4be6-8bf9-dbad0d605fd9","Type":"ContainerStarted","Data":"6060760154d550f41c733ac486f74ba7c0f3ab157939672b0901a6fe2ae96824"}
Sep 19 10:49:43.550693 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:43.550667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59" event={"ID":"c47630c3-6941-4929-8c74-a79aa06e92a8","Type":"ContainerStarted","Data":"d786fd67157b566e6d69947fefb5744b91a70bfe8c7e39b196e5181f7cef8981"}
Sep 19 10:49:43.550826 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:43.550705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59" event={"ID":"c47630c3-6941-4929-8c74-a79aa06e92a8","Type":"ContainerStarted","Data":"1539ba59740558c31ad41c18f0eb69dabeae48e53568467dcf3f162562008c1b"}
Sep 19 10:49:43.586613 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:43.586549    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59" podStartSLOduration=1.5865252600000002 podStartE2EDuration="1.58652526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:49:43.583542266 +0000 UTC m=+828.242287427" watchObservedRunningTime="2026-09-19 10:49:43.58652526 +0000 UTC m=+828.245270418"
Sep 19 10:49:43.938585 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:43.938551    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:43.938838 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:43.938814    2580 patch_prober.go:28] interesting pod/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 19 10:49:43.938892 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:43.938870    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 19 10:49:44.938316 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:44.938273    2580 patch_prober.go:28] interesting pod/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 19 10:49:44.938719 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:44.938346    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 19 10:49:45.938565 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:45.938526    2580 patch_prober.go:28] interesting pod/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 19 10:49:45.938968 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:45.938583    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 19 10:49:46.817970 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:46.817934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:46.938690 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:46.938658    2580 patch_prober.go:28] interesting pod/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 19 10:49:46.939062 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:46.938714    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 19 10:49:47.938554 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:47.938459    2580 patch_prober.go:28] interesting pod/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 19 10:49:47.938554 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:47.938514    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 19 10:49:48.231703 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:48.231625    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:48.835979 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:48.835934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:49.024941 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:49.024911    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:49.025427 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:49.025273    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:49.026380 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:49.026362    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:49:49.562359 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:49.562320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:50.309151 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:50.309111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:51.026859 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:51.026824    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:51.746146 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:51.746111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:52.522503 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:52.522454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:53.013435 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:53.013396    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:53.013945 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:53.013464    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:53.026199 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:53.026174    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:53.187139 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:53.187098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:53.622097 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:53.622059    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 10:49:53.908680 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:53.908564    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:55.007254 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.007220    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:55.521430 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.521396    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"]
Sep 19 10:49:55.521705 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.521648    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68" podUID="130e6f58-a3b9-4601-a39d-9628b9837801" containerName="maas-api" containerID="cri-o://edd35e22a3df70fa2bab0cf3df3e884ba4e062ee8e6ea723bf76e5547a05a425" gracePeriod=30
Sep 19 10:49:55.543582 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.543531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:55.615503 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.615467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:55.680901 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.680865    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:55.742534 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.742500    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:55.779022 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.778944    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:55.841471 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.841434    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/130e6f58-a3b9-4601-a39d-9628b9837801-openshift-service-ca\") pod \"130e6f58-a3b9-4601-a39d-9628b9837801\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") "
Sep 19 10:49:55.841668 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.841512    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxs4r\" (UniqueName: \"kubernetes.io/projected/130e6f58-a3b9-4601-a39d-9628b9837801-kube-api-access-xxs4r\") pod \"130e6f58-a3b9-4601-a39d-9628b9837801\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") "
Sep 19 10:49:55.841668 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.841571    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-maas-api-tls\") pod \"130e6f58-a3b9-4601-a39d-9628b9837801\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") "
Sep 19 10:49:55.841668 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.841596    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-metrics-certs\") pod \"130e6f58-a3b9-4601-a39d-9628b9837801\" (UID: \"130e6f58-a3b9-4601-a39d-9628b9837801\") "
Sep 19 10:49:55.842984 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.842951    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"]
Sep 19 10:49:55.843237 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.843210    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerName="istio-proxy" containerID="cri-o://6060760154d550f41c733ac486f74ba7c0f3ab157939672b0901a6fe2ae96824" gracePeriod=30
Sep 19 10:49:55.846635 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.844666    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "130e6f58-a3b9-4601-a39d-9628b9837801" (UID: "130e6f58-a3b9-4601-a39d-9628b9837801"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:49:55.846635 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.845044    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130e6f58-a3b9-4601-a39d-9628b9837801-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "130e6f58-a3b9-4601-a39d-9628b9837801" (UID: "130e6f58-a3b9-4601-a39d-9628b9837801"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:49:55.850675 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.850127    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "130e6f58-a3b9-4601-a39d-9628b9837801" (UID: "130e6f58-a3b9-4601-a39d-9628b9837801"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:49:55.850675 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.850243    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130e6f58-a3b9-4601-a39d-9628b9837801-kube-api-access-xxs4r" (OuterVolumeSpecName: "kube-api-access-xxs4r") pod "130e6f58-a3b9-4601-a39d-9628b9837801" (UID: "130e6f58-a3b9-4601-a39d-9628b9837801"). InnerVolumeSpecName "kube-api-access-xxs4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:49:55.867734 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.867695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:55.943199 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.943150    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-maas-api-tls\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:55.943199 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.943190    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/130e6f58-a3b9-4601-a39d-9628b9837801-metrics-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:55.943199 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.943205    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/130e6f58-a3b9-4601-a39d-9628b9837801-openshift-service-ca\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:55.943556 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:55.943220    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxs4r\" (UniqueName: \"kubernetes.io/projected/130e6f58-a3b9-4601-a39d-9628b9837801-kube-api-access-xxs4r\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:49:56.189599 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.189552    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:56.297235 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.297190    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:56.604981 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.604905    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"]
Sep 19 10:49:56.605264 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.605251    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130e6f58-a3b9-4601-a39d-9628b9837801" containerName="maas-api"
Sep 19 10:49:56.605315 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.605265    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="130e6f58-a3b9-4601-a39d-9628b9837801" containerName="maas-api"
Sep 19 10:49:56.605356 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.605324    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="130e6f58-a3b9-4601-a39d-9628b9837801" containerName="maas-api"
Sep 19 10:49:56.605356 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.605333    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="89e2ff36-97ab-440f-bcb3-cdfc65699add" containerName="istio-proxy"
Sep 19 10:49:56.609263 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.609244    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.611932 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.611910    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d303059f-openshift-default-dockercfg-9dsxq\""
Sep 19 10:49:56.634901 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.634864    2580 generic.go:356] "Generic (PLEG): container finished" podID="130e6f58-a3b9-4601-a39d-9628b9837801" containerID="edd35e22a3df70fa2bab0cf3df3e884ba4e062ee8e6ea723bf76e5547a05a425" exitCode=0
Sep 19 10:49:56.635057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.634911    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"
Sep 19 10:49:56.635057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.634950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68" event={"ID":"130e6f58-a3b9-4601-a39d-9628b9837801","Type":"ContainerDied","Data":"edd35e22a3df70fa2bab0cf3df3e884ba4e062ee8e6ea723bf76e5547a05a425"}
Sep 19 10:49:56.635057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.635002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68" event={"ID":"130e6f58-a3b9-4601-a39d-9628b9837801","Type":"ContainerDied","Data":"7f602c5e24d175c570949c2b232df5d787153060bb9f42bbfbfa68107fdf01c2"}
Sep 19 10:49:56.635057 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.635026    2580 scope.go:117] "RemoveContainer" containerID="edd35e22a3df70fa2bab0cf3df3e884ba4e062ee8e6ea723bf76e5547a05a425"
Sep 19 10:49:56.637845 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.637816    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"]
Sep 19 10:49:56.645471 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.645449    2580 scope.go:117] "RemoveContainer" containerID="edd35e22a3df70fa2bab0cf3df3e884ba4e062ee8e6ea723bf76e5547a05a425"
Sep 19 10:49:56.645784 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:49:56.645747    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd35e22a3df70fa2bab0cf3df3e884ba4e062ee8e6ea723bf76e5547a05a425\": container with ID starting with edd35e22a3df70fa2bab0cf3df3e884ba4e062ee8e6ea723bf76e5547a05a425 not found: ID does not exist" containerID="edd35e22a3df70fa2bab0cf3df3e884ba4e062ee8e6ea723bf76e5547a05a425"
Sep 19 10:49:56.645902 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.645791    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd35e22a3df70fa2bab0cf3df3e884ba4e062ee8e6ea723bf76e5547a05a425"} err="failed to get container status \"edd35e22a3df70fa2bab0cf3df3e884ba4e062ee8e6ea723bf76e5547a05a425\": rpc error: code = NotFound desc = could not find container \"edd35e22a3df70fa2bab0cf3df3e884ba4e062ee8e6ea723bf76e5547a05a425\": container with ID starting with edd35e22a3df70fa2bab0cf3df3e884ba4e062ee8e6ea723bf76e5547a05a425 not found: ID does not exist"
Sep 19 10:49:56.650579 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.650560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-credential-socket\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.650669 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.650592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-workload-certs\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.650669 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.650613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89pw8\" (UniqueName: \"kubernetes.io/projected/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-kube-api-access-89pw8\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.650669 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.650653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-data\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.650783 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.650746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istiod-ca-cert\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.650820 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.650792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-workload-socket\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.650820 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.650811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-envoy\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.650895 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.650870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-podinfo\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.650895 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.650887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-token\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.665453 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.665421    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:56.699883 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.699849    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"]
Sep 19 10:49:56.704789 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.704748    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-18e8cec5-f5858f664-klq68"]
Sep 19 10:49:56.751463 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.751414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-workload-certs\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.751463 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.751463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89pw8\" (UniqueName: \"kubernetes.io/projected/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-kube-api-access-89pw8\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.751698 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.751487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-data\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.751698 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.751548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istiod-ca-cert\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.751698 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.751564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-workload-socket\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.751698 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.751584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-envoy\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.751698 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.751619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-podinfo\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.751698 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.751640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-token\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.751698 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.751695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-credential-socket\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.752073 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.751932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-workload-certs\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.752073 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.751995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-data\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.752073 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.752060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-workload-socket\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.752186 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.752105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-credential-socket\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.752364 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.752341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istiod-ca-cert\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.754114 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.754083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-envoy\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.754359 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.754342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-podinfo\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.758893 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.758866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-token\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.759000 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.758916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89pw8\" (UniqueName: \"kubernetes.io/projected/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-kube-api-access-89pw8\") pod \"e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") " pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:56.919525 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:56.919398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:57.004447 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:57.004398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:57.203154 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:57.203129    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"]
Sep 19 10:49:57.205700 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:49:57.205671    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247f7a92_6a26_4f8e_9e71_19ce08ab7b76.slice/crio-88241760168d0213bccce66522c798f3d592b1a8789eddb4abd2b74665dd3f1c WatchSource:0}: Error finding container 88241760168d0213bccce66522c798f3d592b1a8789eddb4abd2b74665dd3f1c: Status 404 returned error can't find the container with id 88241760168d0213bccce66522c798f3d592b1a8789eddb4abd2b74665dd3f1c
Sep 19 10:49:57.208077 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:57.208042    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:57.208158 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:57.208132    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:57.208198 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:57.208161    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:49:57.371706 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:57.371669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:57.642064 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:57.641952    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" event={"ID":"247f7a92-6a26-4f8e-9e71-19ce08ab7b76","Type":"ContainerStarted","Data":"4c7318add840cc43540fd0a445bda05759498c58998266cf5aee75831d29367e"}
Sep 19 10:49:57.642064 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:57.642014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" event={"ID":"247f7a92-6a26-4f8e-9e71-19ce08ab7b76","Type":"ContainerStarted","Data":"88241760168d0213bccce66522c798f3d592b1a8789eddb4abd2b74665dd3f1c"}
Sep 19 10:49:57.662179 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:57.662130    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" podStartSLOduration=1.662113104 podStartE2EDuration="1.662113104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:49:57.65960563 +0000 UTC m=+842.318350792" watchObservedRunningTime="2026-09-19 10:49:57.662113104 +0000 UTC m=+842.320858263"
Sep 19 10:49:57.713577 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:57.713535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:57.856770 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:57.856724    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130e6f58-a3b9-4601-a39d-9628b9837801" path="/var/lib/kubelet/pods/130e6f58-a3b9-4601-a39d-9628b9837801/volumes"
Sep 19 10:49:57.919620 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:57.919521    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:58.419921 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:58.419875    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:58.924998 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:58.924967    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:58.990925 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:58.990895    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:59.090418 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:59.090377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:59.191135 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:59.191104    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:49:59.652804 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:59.652694    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:59.653955 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:59.653938    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:49:59.913373 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:59.913286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:49:59.949061 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:49:59.949024    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:00.656339 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:00.656301    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:00.712235 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:00.712199    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:00.853388 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:00.853355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:01.107004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.106969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:01.122553 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.122526    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:50:01.195052 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195015    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istiod-ca-cert\") pod \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") "
Sep 19 10:50:01.195052 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-token\") pod \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") "
Sep 19 10:50:01.195305 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195086    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzprw\" (UniqueName: \"kubernetes.io/projected/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-kube-api-access-jzprw\") pod \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") "
Sep 19 10:50:01.195305 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195125    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-credential-socket\") pod \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") "
Sep 19 10:50:01.195305 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195152    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-workload-socket\") pod \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") "
Sep 19 10:50:01.195305 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195202    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-podinfo\") pod \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") "
Sep 19 10:50:01.195305 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195264    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-data\") pod \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") "
Sep 19 10:50:01.195305 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195298    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-workload-certs\") pod \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") "
Sep 19 10:50:01.195611 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195323    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-envoy\") pod \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\" (UID: \"a98a631a-d59d-4be6-8bf9-dbad0d605fd9\") "
Sep 19 10:50:01.195611 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195410    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a98a631a-d59d-4be6-8bf9-dbad0d605fd9" (UID: "a98a631a-d59d-4be6-8bf9-dbad0d605fd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:50:01.195611 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195579    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a98a631a-d59d-4be6-8bf9-dbad0d605fd9" (UID: "a98a631a-d59d-4be6-8bf9-dbad0d605fd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:50:01.195814 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195641    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-data" (OuterVolumeSpecName: "istio-data") pod "a98a631a-d59d-4be6-8bf9-dbad0d605fd9" (UID: "a98a631a-d59d-4be6-8bf9-dbad0d605fd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:50:01.195814 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195683    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-workload-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:01.195814 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195701    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istiod-ca-cert\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:01.195814 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.195712    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a98a631a-d59d-4be6-8bf9-dbad0d605fd9" (UID: "a98a631a-d59d-4be6-8bf9-dbad0d605fd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:50:01.196258 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.196143    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a98a631a-d59d-4be6-8bf9-dbad0d605fd9" (UID: "a98a631a-d59d-4be6-8bf9-dbad0d605fd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:50:01.197578 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.197554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a98a631a-d59d-4be6-8bf9-dbad0d605fd9" (UID: "a98a631a-d59d-4be6-8bf9-dbad0d605fd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 10:50:01.197681 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.197599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "a98a631a-d59d-4be6-8bf9-dbad0d605fd9" (UID: "a98a631a-d59d-4be6-8bf9-dbad0d605fd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:50:01.198279 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.198254    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a98a631a-d59d-4be6-8bf9-dbad0d605fd9" (UID: "a98a631a-d59d-4be6-8bf9-dbad0d605fd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:50:01.198279 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.198263    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-kube-api-access-jzprw" (OuterVolumeSpecName: "kube-api-access-jzprw") pod "a98a631a-d59d-4be6-8bf9-dbad0d605fd9" (UID: "a98a631a-d59d-4be6-8bf9-dbad0d605fd9"). InnerVolumeSpecName "kube-api-access-jzprw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:50:01.203246 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.203218    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:01.203870 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.203847    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"]
Sep 19 10:50:01.204272 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.204259    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerName="istio-proxy"
Sep 19 10:50:01.204340 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.204274    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerName="istio-proxy"
Sep 19 10:50:01.204399 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.204359    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerName="istio-proxy"
Sep 19 10:50:01.204399 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.204367    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerName="istio-proxy"
Sep 19 10:50:01.207610 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.207591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"
Sep 19 10:50:01.209510 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.209489    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d303059f\""
Sep 19 10:50:01.209611 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.209490    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d303059f-dockercfg-cnsb9\""
Sep 19 10:50:01.217305 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.217184    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"]
Sep 19 10:50:01.262248 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.262212    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"]
Sep 19 10:50:01.262616 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.262598    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerName="istio-proxy"
Sep 19 10:50:01.262616 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.262616    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerName="istio-proxy"
Sep 19 10:50:01.265854 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.265831    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"
Sep 19 10:50:01.274217 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.274190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"]
Sep 19 10:50:01.296271 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.296227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f914ab-e8c7-4e96-922a-b6a838980465-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd\" (UID: \"75f914ab-e8c7-4e96-922a-b6a838980465\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"
Sep 19 10:50:01.296477 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.296295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beda4ff5-3060-4862-8a90-b54b8806f09d-plugins-config-volume\") pod \"payload-processing-e2e-ait-d303059f-55757f78ff-76l8j\" (UID: \"beda4ff5-3060-4862-8a90-b54b8806f09d\") " pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"
Sep 19 10:50:01.296477 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.296354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtq45\" (UniqueName: \"kubernetes.io/projected/beda4ff5-3060-4862-8a90-b54b8806f09d-kube-api-access-mtq45\") pod \"payload-processing-e2e-ait-d303059f-55757f78ff-76l8j\" (UID: \"beda4ff5-3060-4862-8a90-b54b8806f09d\") " pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"
Sep 19 10:50:01.296614 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.296552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvqnh\" (UniqueName: \"kubernetes.io/projected/75f914ab-e8c7-4e96-922a-b6a838980465-kube-api-access-qvqnh\") pod \"payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd\" (UID: \"75f914ab-e8c7-4e96-922a-b6a838980465\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"
Sep 19 10:50:01.296670 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.296618    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-token\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:01.296670 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.296632    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzprw\" (UniqueName: \"kubernetes.io/projected/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-kube-api-access-jzprw\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:01.296670 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.296645    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-credential-socket\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:01.296839 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.296669    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-workload-socket\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:01.296839 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.296686    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-podinfo\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:01.296839 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.296697    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-data\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:01.296839 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.296709    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a98a631a-d59d-4be6-8bf9-dbad0d605fd9-istio-envoy\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:01.398062 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.398028    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvqnh\" (UniqueName: \"kubernetes.io/projected/75f914ab-e8c7-4e96-922a-b6a838980465-kube-api-access-qvqnh\") pod \"payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd\" (UID: \"75f914ab-e8c7-4e96-922a-b6a838980465\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"
Sep 19 10:50:01.398271 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.398110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f914ab-e8c7-4e96-922a-b6a838980465-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd\" (UID: \"75f914ab-e8c7-4e96-922a-b6a838980465\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"
Sep 19 10:50:01.398271 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.398151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beda4ff5-3060-4862-8a90-b54b8806f09d-plugins-config-volume\") pod \"payload-processing-e2e-ait-d303059f-55757f78ff-76l8j\" (UID: \"beda4ff5-3060-4862-8a90-b54b8806f09d\") " pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"
Sep 19 10:50:01.398271 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.398192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtq45\" (UniqueName: \"kubernetes.io/projected/beda4ff5-3060-4862-8a90-b54b8806f09d-kube-api-access-mtq45\") pod \"payload-processing-e2e-ait-d303059f-55757f78ff-76l8j\" (UID: \"beda4ff5-3060-4862-8a90-b54b8806f09d\") " pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"
Sep 19 10:50:01.398887 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.398865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beda4ff5-3060-4862-8a90-b54b8806f09d-plugins-config-volume\") pod \"payload-processing-e2e-ait-d303059f-55757f78ff-76l8j\" (UID: \"beda4ff5-3060-4862-8a90-b54b8806f09d\") " pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"
Sep 19 10:50:01.398935 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.398885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f914ab-e8c7-4e96-922a-b6a838980465-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd\" (UID: \"75f914ab-e8c7-4e96-922a-b6a838980465\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"
Sep 19 10:50:01.405623 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.405597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtq45\" (UniqueName: \"kubernetes.io/projected/beda4ff5-3060-4862-8a90-b54b8806f09d-kube-api-access-mtq45\") pod \"payload-processing-e2e-ait-d303059f-55757f78ff-76l8j\" (UID: \"beda4ff5-3060-4862-8a90-b54b8806f09d\") " pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"
Sep 19 10:50:01.405714 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.405688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvqnh\" (UniqueName: \"kubernetes.io/projected/75f914ab-e8c7-4e96-922a-b6a838980465-kube-api-access-qvqnh\") pod \"payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd\" (UID: \"75f914ab-e8c7-4e96-922a-b6a838980465\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"
Sep 19 10:50:01.524202 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.524112    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"
Sep 19 10:50:01.576782 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.576720    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"
Sep 19 10:50:01.617368 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.617329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:01.673253 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.673216    2580 generic.go:356] "Generic (PLEG): container finished" podID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" containerID="6060760154d550f41c733ac486f74ba7c0f3ab157939672b0901a6fe2ae96824" exitCode=0
Sep 19 10:50:01.673840 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.673817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" event={"ID":"a98a631a-d59d-4be6-8bf9-dbad0d605fd9","Type":"ContainerDied","Data":"6060760154d550f41c733ac486f74ba7c0f3ab157939672b0901a6fe2ae96824"}
Sep 19 10:50:01.673911 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.673853    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"
Sep 19 10:50:01.673911 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.673869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n" event={"ID":"a98a631a-d59d-4be6-8bf9-dbad0d605fd9","Type":"ContainerDied","Data":"0e01bff58af337c01b52f52bb17e14a9ed42aac8a95a7ba63158c9ddebc02a11"}
Sep 19 10:50:01.673911 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.673900    2580 scope.go:117] "RemoveContainer" containerID="6060760154d550f41c733ac486f74ba7c0f3ab157939672b0901a6fe2ae96824"
Sep 19 10:50:01.684117 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.684098    2580 scope.go:117] "RemoveContainer" containerID="b053032dc46bfe43ddcd17cd6af8c494cd1d9f6f454aec6a21d874550cb9bebe"
Sep 19 10:50:01.699090 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.698994    2580 scope.go:117] "RemoveContainer" containerID="6060760154d550f41c733ac486f74ba7c0f3ab157939672b0901a6fe2ae96824"
Sep 19 10:50:01.699653 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:50:01.699482    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6060760154d550f41c733ac486f74ba7c0f3ab157939672b0901a6fe2ae96824\": container with ID starting with 6060760154d550f41c733ac486f74ba7c0f3ab157939672b0901a6fe2ae96824 not found: ID does not exist" containerID="6060760154d550f41c733ac486f74ba7c0f3ab157939672b0901a6fe2ae96824"
Sep 19 10:50:01.699653 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.699526    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6060760154d550f41c733ac486f74ba7c0f3ab157939672b0901a6fe2ae96824"} err="failed to get container status \"6060760154d550f41c733ac486f74ba7c0f3ab157939672b0901a6fe2ae96824\": rpc error: code = NotFound desc = could not find container \"6060760154d550f41c733ac486f74ba7c0f3ab157939672b0901a6fe2ae96824\": container with ID starting with 6060760154d550f41c733ac486f74ba7c0f3ab157939672b0901a6fe2ae96824 not found: ID does not exist"
Sep 19 10:50:01.699653 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.699553    2580 scope.go:117] "RemoveContainer" containerID="b053032dc46bfe43ddcd17cd6af8c494cd1d9f6f454aec6a21d874550cb9bebe"
Sep 19 10:50:01.700076 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:50:01.700054    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b053032dc46bfe43ddcd17cd6af8c494cd1d9f6f454aec6a21d874550cb9bebe\": container with ID starting with b053032dc46bfe43ddcd17cd6af8c494cd1d9f6f454aec6a21d874550cb9bebe not found: ID does not exist" containerID="b053032dc46bfe43ddcd17cd6af8c494cd1d9f6f454aec6a21d874550cb9bebe"
Sep 19 10:50:01.700193 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.700080    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b053032dc46bfe43ddcd17cd6af8c494cd1d9f6f454aec6a21d874550cb9bebe"} err="failed to get container status \"b053032dc46bfe43ddcd17cd6af8c494cd1d9f6f454aec6a21d874550cb9bebe\": rpc error: code = NotFound desc = could not find container \"b053032dc46bfe43ddcd17cd6af8c494cd1d9f6f454aec6a21d874550cb9bebe\": container with ID starting with b053032dc46bfe43ddcd17cd6af8c494cd1d9f6f454aec6a21d874550cb9bebe not found: ID does not exist"
Sep 19 10:50:01.700293 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.700273    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"]
Sep 19 10:50:01.702523 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.702488    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-18e8cec5-gw-openshift-default-f7985c8f6-cbx9n"]
Sep 19 10:50:01.797391 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.797367    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"]
Sep 19 10:50:01.799497 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:50:01.799472    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f914ab_e8c7_4e96_922a_b6a838980465.slice/crio-3466b460399c118addf4c8d7345648475717b06c8f0115d1cf235f6ca6f97ce5 WatchSource:0}: Error finding container 3466b460399c118addf4c8d7345648475717b06c8f0115d1cf235f6ca6f97ce5: Status 404 returned error can't find the container with id 3466b460399c118addf4c8d7345648475717b06c8f0115d1cf235f6ca6f97ce5
Sep 19 10:50:01.855992 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.855956    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a98a631a-d59d-4be6-8bf9-dbad0d605fd9" path="/var/lib/kubelet/pods/a98a631a-d59d-4be6-8bf9-dbad0d605fd9/volumes"
Sep 19 10:50:01.865214 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:01.865187    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"]
Sep 19 10:50:01.867088 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:50:01.867067    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeda4ff5_3060_4862_8a90_b54b8806f09d.slice/crio-4cf7003e8dc077e2e59c61ce6fd2e08b2baec32e301561d0e0c5cad4cc257b97 WatchSource:0}: Error finding container 4cf7003e8dc077e2e59c61ce6fd2e08b2baec32e301561d0e0c5cad4cc257b97: Status 404 returned error can't find the container with id 4cf7003e8dc077e2e59c61ce6fd2e08b2baec32e301561d0e0c5cad4cc257b97
Sep 19 10:50:02.341343 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:02.341303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:02.685490 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:02.685448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd" event={"ID":"75f914ab-e8c7-4e96-922a-b6a838980465","Type":"ContainerStarted","Data":"3466b460399c118addf4c8d7345648475717b06c8f0115d1cf235f6ca6f97ce5"}
Sep 19 10:50:02.687121 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:02.687091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j" event={"ID":"beda4ff5-3060-4862-8a90-b54b8806f09d","Type":"ContainerStarted","Data":"4cf7003e8dc077e2e59c61ce6fd2e08b2baec32e301561d0e0c5cad4cc257b97"}
Sep 19 10:50:03.110214 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:03.110156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:03.702593 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:03.702555    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j" event={"ID":"beda4ff5-3060-4862-8a90-b54b8806f09d","Type":"ContainerStarted","Data":"b5907a6fe18129bd46acba21b1f55ef36f0bfd40daa5197f6720b4f581271f0b"}
Sep 19 10:50:03.703106 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:03.702653    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"
Sep 19 10:50:03.704349 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:03.704322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd" event={"ID":"75f914ab-e8c7-4e96-922a-b6a838980465","Type":"ContainerStarted","Data":"696c37fa2a174fbd6af2b0628680eb12267268c58d7535c66c92d042f0019404"}
Sep 19 10:50:03.704485 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:03.704465    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"
Sep 19 10:50:03.717485 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:03.717428    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j" podStartSLOduration=1.492992347 podStartE2EDuration="2.71741241s" totalImagesPullingTime="1.224420063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:50:03.71691503 +0000 UTC m=+848.375660194" watchObservedRunningTime="2026-09-19 10:50:03.71741241 +0000 UTC m=+848.376157570"
Sep 19 10:50:03.720977 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:03.720950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:03.732599 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:03.732542    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd" podStartSLOduration=1.4378837 podStartE2EDuration="2.732521295s" totalImagesPullingTime="1.294637595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:50:03.730376223 +0000 UTC m=+848.389121397" watchObservedRunningTime="2026-09-19 10:50:03.732521295 +0000 UTC m=+848.391266456"
Sep 19 10:50:14.711980 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:14.711947    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"
Sep 19 10:50:14.712432 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:14.712022    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"
Sep 19 10:50:18.111402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:18.111351    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:50:19.793413 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:19.793374    2580 generic.go:356] "Generic (PLEG): container finished" podID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerID="4c7318add840cc43540fd0a445bda05759498c58998266cf5aee75831d29367e" exitCode=0
Sep 19 10:50:19.793860 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:19.793449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" event={"ID":"247f7a92-6a26-4f8e-9e71-19ce08ab7b76","Type":"ContainerDied","Data":"4c7318add840cc43540fd0a445bda05759498c58998266cf5aee75831d29367e"}
Sep 19 10:50:19.793860 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:19.793854    2580 scope.go:117] "RemoveContainer" containerID="4c7318add840cc43540fd0a445bda05759498c58998266cf5aee75831d29367e"
Sep 19 10:50:19.795106 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:19.795080    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:50:19.795162 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:19.795154    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:50:19.795197 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:19.795184    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:50:20.801417 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:20.801374    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" event={"ID":"247f7a92-6a26-4f8e-9e71-19ce08ab7b76","Type":"ContainerStarted","Data":"77dc45001afbb089be35557ecef2c948dc22b6a034e14ea7c0d9cd74c68ff7f5"}
Sep 19 10:50:20.920080 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:20.920041    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:50:20.920304 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:20.920282    2580 patch_prober.go:28] interesting pod/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 19 10:50:20.920349 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:20.920333    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 19 10:50:21.920351 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:21.920307    2580 patch_prober.go:28] interesting pod/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 19 10:50:21.920812 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:21.920371    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 19 10:50:22.610297 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:22.610252    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:22.920719 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:22.920628    2580 patch_prober.go:28] interesting pod/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 19 10:50:22.920719 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:22.920688    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 19 10:50:23.228166 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:23.228129    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:23.920727 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:23.920688    2580 patch_prober.go:28] interesting pod/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 19 10:50:23.921215 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:23.920742    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 19 10:50:23.999408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:23.999364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:24.624282 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:24.624244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:24.920165 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:24.920069    2580 patch_prober.go:28] interesting pod/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 19 10:50:24.920329 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:24.920165    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 19 10:50:25.306638 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:25.306599    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:25.920597 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:25.920557    2580 patch_prober.go:28] interesting pod/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 19 10:50:25.920844 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:25.920619    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 19 10:50:26.038037 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:26.037995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:26.714440 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:26.714404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:26.924795 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:26.924739    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:50:26.925277 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:26.925253    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:50:26.925956 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:26.925938    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:50:27.514520 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:27.514477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:28.131652 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:28.131621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:28.801459 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:28.801424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:29.205405 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.205377    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"]
Sep 19 10:50:29.210113 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.210093    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.212269 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.212245    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1907c1\"/\"kube-root-ca.crt\""
Sep 19 10:50:29.212978 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.212958    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1907c1\"/\"openshift-service-ca.crt\""
Sep 19 10:50:29.213091 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.213000    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1907c1\"/\"default-dockercfg-d8flv\""
Sep 19 10:50:29.213091 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.213000    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1907c1\"/\"simulator-w0-1907c1-kserve-self-signed-certs\""
Sep 19 10:50:29.216064 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.216043    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"]
Sep 19 10:50:29.221440 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.221411    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:29.355159 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.355120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-dshm\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.355342 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.355170    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10945f3e-0a8b-413e-b27c-76b1255e1704-tls-certs\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.355342 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.355199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hq4f\" (UniqueName: \"kubernetes.io/projected/10945f3e-0a8b-413e-b27c-76b1255e1704-kube-api-access-2hq4f\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.355342 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.355238    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-model-cache\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.355460 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.355339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-home\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.355460 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.355364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-tmp-dir\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.455968 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.455890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-home\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.455968 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.455923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-tmp-dir\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.456173 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.455985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-dshm\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.456173 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.456015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10945f3e-0a8b-413e-b27c-76b1255e1704-tls-certs\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.456173 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.456032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hq4f\" (UniqueName: \"kubernetes.io/projected/10945f3e-0a8b-413e-b27c-76b1255e1704-kube-api-access-2hq4f\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.456173 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.456059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-model-cache\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.456333 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.456307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-home\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.456401 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.456366    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-tmp-dir\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.456459 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.456410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-model-cache\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.458583 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.458529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-dshm\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.459169 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.459144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10945f3e-0a8b-413e-b27c-76b1255e1704-tls-certs\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.463659 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.463631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hq4f\" (UniqueName: \"kubernetes.io/projected/10945f3e-0a8b-413e-b27c-76b1255e1704-kube-api-access-2hq4f\") pod \"simulator-w0-1907c1-kserve-7f8b595564-h7gk6\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") " pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.520594 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.520558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:29.616623 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.616570    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:29.791994 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.791964    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"]
Sep 19 10:50:29.794163 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:50:29.794139    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10945f3e_0a8b_413e_b27c_76b1255e1704.slice/crio-d0e2a9c5bfadcaaab52ef9601084a6fce386d6324b37eaebe38d1e6ca658b283 WatchSource:0}: Error finding container d0e2a9c5bfadcaaab52ef9601084a6fce386d6324b37eaebe38d1e6ca658b283: Status 404 returned error can't find the container with id d0e2a9c5bfadcaaab52ef9601084a6fce386d6324b37eaebe38d1e6ca658b283
Sep 19 10:50:29.858496 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:29.857801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6" event={"ID":"10945f3e-0a8b-413e-b27c-76b1255e1704","Type":"ContainerStarted","Data":"d0e2a9c5bfadcaaab52ef9601084a6fce386d6324b37eaebe38d1e6ca658b283"}
Sep 19 10:50:30.355326 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:30.355286    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:30.866580 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:30.866542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6" event={"ID":"10945f3e-0a8b-413e-b27c-76b1255e1704","Type":"ContainerStarted","Data":"1cb25cb054c356697c8f51375b7913f9e67b9fc70b699f4b8ace66b880c9d6bf"}
Sep 19 10:50:30.881604 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:30.881549    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6" podStartSLOduration=1.881527824 podStartE2EDuration="1.881527824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:50:30.880905067 +0000 UTC m=+875.539650232" watchObservedRunningTime="2026-09-19 10:50:30.881527824 +0000 UTC m=+875.540272983"
Sep 19 10:50:30.942356 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:30.942317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:30.969890 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:30.969848    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"]
Sep 19 10:50:30.970178 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:30.970152    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j" podUID="beda4ff5-3060-4862-8a90-b54b8806f09d" containerName="payload-processing" containerID="cri-o://b5907a6fe18129bd46acba21b1f55ef36f0bfd40daa5197f6720b4f581271f0b" gracePeriod=30
Sep 19 10:50:30.977839 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:30.977804    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"]
Sep 19 10:50:30.978579 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:30.978515    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd" podUID="75f914ab-e8c7-4e96-922a-b6a838980465" containerName="payload-pre-processing" containerID="cri-o://696c37fa2a174fbd6af2b0628680eb12267268c58d7535c66c92d042f0019404" gracePeriod=30
Sep 19 10:50:31.009230 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.009198    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:31.110095 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.110028    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:31.162797 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.162353    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:31.228235 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.228206    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"
Sep 19 10:50:31.250024 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.249994    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"
Sep 19 10:50:31.374700 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.374668    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtq45\" (UniqueName: \"kubernetes.io/projected/beda4ff5-3060-4862-8a90-b54b8806f09d-kube-api-access-mtq45\") pod \"beda4ff5-3060-4862-8a90-b54b8806f09d\" (UID: \"beda4ff5-3060-4862-8a90-b54b8806f09d\") "
Sep 19 10:50:31.374700 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.374705    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvqnh\" (UniqueName: \"kubernetes.io/projected/75f914ab-e8c7-4e96-922a-b6a838980465-kube-api-access-qvqnh\") pod \"75f914ab-e8c7-4e96-922a-b6a838980465\" (UID: \"75f914ab-e8c7-4e96-922a-b6a838980465\") "
Sep 19 10:50:31.375257 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.374877    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beda4ff5-3060-4862-8a90-b54b8806f09d-plugins-config-volume\") pod \"beda4ff5-3060-4862-8a90-b54b8806f09d\" (UID: \"beda4ff5-3060-4862-8a90-b54b8806f09d\") "
Sep 19 10:50:31.375257 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.374931    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f914ab-e8c7-4e96-922a-b6a838980465-plugins-config-volume\") pod \"75f914ab-e8c7-4e96-922a-b6a838980465\" (UID: \"75f914ab-e8c7-4e96-922a-b6a838980465\") "
Sep 19 10:50:31.375257 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.375244    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beda4ff5-3060-4862-8a90-b54b8806f09d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "beda4ff5-3060-4862-8a90-b54b8806f09d" (UID: "beda4ff5-3060-4862-8a90-b54b8806f09d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:50:31.375370 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.375307    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f914ab-e8c7-4e96-922a-b6a838980465-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "75f914ab-e8c7-4e96-922a-b6a838980465" (UID: "75f914ab-e8c7-4e96-922a-b6a838980465"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:50:31.377131 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.377108    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f914ab-e8c7-4e96-922a-b6a838980465-kube-api-access-qvqnh" (OuterVolumeSpecName: "kube-api-access-qvqnh") pod "75f914ab-e8c7-4e96-922a-b6a838980465" (UID: "75f914ab-e8c7-4e96-922a-b6a838980465"). InnerVolumeSpecName "kube-api-access-qvqnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:50:31.377217 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.377142    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beda4ff5-3060-4862-8a90-b54b8806f09d-kube-api-access-mtq45" (OuterVolumeSpecName: "kube-api-access-mtq45") pod "beda4ff5-3060-4862-8a90-b54b8806f09d" (UID: "beda4ff5-3060-4862-8a90-b54b8806f09d"). InnerVolumeSpecName "kube-api-access-mtq45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:50:31.476037 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.476000    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beda4ff5-3060-4862-8a90-b54b8806f09d-plugins-config-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:31.476037 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.476032    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f914ab-e8c7-4e96-922a-b6a838980465-plugins-config-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:31.476037 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.476042    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtq45\" (UniqueName: \"kubernetes.io/projected/beda4ff5-3060-4862-8a90-b54b8806f09d-kube-api-access-mtq45\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:31.476266 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.476050    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvqnh\" (UniqueName: \"kubernetes.io/projected/75f914ab-e8c7-4e96-922a-b6a838980465-kube-api-access-qvqnh\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:31.553650 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.553619    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:31.873895 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.873862    2580 generic.go:356] "Generic (PLEG): container finished" podID="beda4ff5-3060-4862-8a90-b54b8806f09d" containerID="b5907a6fe18129bd46acba21b1f55ef36f0bfd40daa5197f6720b4f581271f0b" exitCode=0
Sep 19 10:50:31.874058 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.873909    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"
Sep 19 10:50:31.874058 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.873955    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j" event={"ID":"beda4ff5-3060-4862-8a90-b54b8806f09d","Type":"ContainerDied","Data":"b5907a6fe18129bd46acba21b1f55ef36f0bfd40daa5197f6720b4f581271f0b"}
Sep 19 10:50:31.874058 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.874014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j" event={"ID":"beda4ff5-3060-4862-8a90-b54b8806f09d","Type":"ContainerDied","Data":"4cf7003e8dc077e2e59c61ce6fd2e08b2baec32e301561d0e0c5cad4cc257b97"}
Sep 19 10:50:31.874058 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.874044    2580 scope.go:117] "RemoveContainer" containerID="b5907a6fe18129bd46acba21b1f55ef36f0bfd40daa5197f6720b4f581271f0b"
Sep 19 10:50:31.875621 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.875594    2580 generic.go:356] "Generic (PLEG): container finished" podID="75f914ab-e8c7-4e96-922a-b6a838980465" containerID="696c37fa2a174fbd6af2b0628680eb12267268c58d7535c66c92d042f0019404" exitCode=0
Sep 19 10:50:31.875817 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.875728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd" event={"ID":"75f914ab-e8c7-4e96-922a-b6a838980465","Type":"ContainerDied","Data":"696c37fa2a174fbd6af2b0628680eb12267268c58d7535c66c92d042f0019404"}
Sep 19 10:50:31.875817 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.875806    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd" event={"ID":"75f914ab-e8c7-4e96-922a-b6a838980465","Type":"ContainerDied","Data":"3466b460399c118addf4c8d7345648475717b06c8f0115d1cf235f6ca6f97ce5"}
Sep 19 10:50:31.876072 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.875886    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"
Sep 19 10:50:31.884008 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.883978    2580 scope.go:117] "RemoveContainer" containerID="b5907a6fe18129bd46acba21b1f55ef36f0bfd40daa5197f6720b4f581271f0b"
Sep 19 10:50:31.884312 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:50:31.884288    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5907a6fe18129bd46acba21b1f55ef36f0bfd40daa5197f6720b4f581271f0b\": container with ID starting with b5907a6fe18129bd46acba21b1f55ef36f0bfd40daa5197f6720b4f581271f0b not found: ID does not exist" containerID="b5907a6fe18129bd46acba21b1f55ef36f0bfd40daa5197f6720b4f581271f0b"
Sep 19 10:50:31.884439 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.884322    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5907a6fe18129bd46acba21b1f55ef36f0bfd40daa5197f6720b4f581271f0b"} err="failed to get container status \"b5907a6fe18129bd46acba21b1f55ef36f0bfd40daa5197f6720b4f581271f0b\": rpc error: code = NotFound desc = could not find container \"b5907a6fe18129bd46acba21b1f55ef36f0bfd40daa5197f6720b4f581271f0b\": container with ID starting with b5907a6fe18129bd46acba21b1f55ef36f0bfd40daa5197f6720b4f581271f0b not found: ID does not exist"
Sep 19 10:50:31.884439 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.884346    2580 scope.go:117] "RemoveContainer" containerID="696c37fa2a174fbd6af2b0628680eb12267268c58d7535c66c92d042f0019404"
Sep 19 10:50:31.892972 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.892950    2580 scope.go:117] "RemoveContainer" containerID="696c37fa2a174fbd6af2b0628680eb12267268c58d7535c66c92d042f0019404"
Sep 19 10:50:31.893585 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:50:31.893554    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"696c37fa2a174fbd6af2b0628680eb12267268c58d7535c66c92d042f0019404\": container with ID starting with 696c37fa2a174fbd6af2b0628680eb12267268c58d7535c66c92d042f0019404 not found: ID does not exist" containerID="696c37fa2a174fbd6af2b0628680eb12267268c58d7535c66c92d042f0019404"
Sep 19 10:50:31.893702 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.893598    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696c37fa2a174fbd6af2b0628680eb12267268c58d7535c66c92d042f0019404"} err="failed to get container status \"696c37fa2a174fbd6af2b0628680eb12267268c58d7535c66c92d042f0019404\": rpc error: code = NotFound desc = could not find container \"696c37fa2a174fbd6af2b0628680eb12267268c58d7535c66c92d042f0019404\": container with ID starting with 696c37fa2a174fbd6af2b0628680eb12267268c58d7535c66c92d042f0019404 not found: ID does not exist"
Sep 19 10:50:31.894246 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.894219    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"]
Sep 19 10:50:31.896306 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.896284    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d303059f-7c47d9fff8-kkqxd"]
Sep 19 10:50:31.905283 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.905257    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"]
Sep 19 10:50:31.909710 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:31.909685    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d303059f-55757f78ff-76l8j"]
Sep 19 10:50:32.265500 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:32.265460    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"]
Sep 19 10:50:32.265710 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:32.265682    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerName="istio-proxy" containerID="cri-o://77dc45001afbb089be35557ecef2c948dc22b6a034e14ea7c0d9cd74c68ff7f5" gracePeriod=30
Sep 19 10:50:32.287161 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:32.287115    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:32.385890 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:32.385851    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:32.724193 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:32.724158    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:32.961066 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:32.961003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:32.999260 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:32.999171    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"]
Sep 19 10:50:32.999559 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:32.999542    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beda4ff5-3060-4862-8a90-b54b8806f09d" containerName="payload-processing"
Sep 19 10:50:32.999638 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:32.999561    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="beda4ff5-3060-4862-8a90-b54b8806f09d" containerName="payload-processing"
Sep 19 10:50:32.999638 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:32.999581    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f914ab-e8c7-4e96-922a-b6a838980465" containerName="payload-pre-processing"
Sep 19 10:50:32.999638 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:32.999590    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f914ab-e8c7-4e96-922a-b6a838980465" containerName="payload-pre-processing"
Sep 19 10:50:32.999814 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:32.999715    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f914ab-e8c7-4e96-922a-b6a838980465" containerName="payload-pre-processing"
Sep 19 10:50:32.999814 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:32.999733    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="beda4ff5-3060-4862-8a90-b54b8806f09d" containerName="payload-processing"
Sep 19 10:50:33.004351 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.004323    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.008917 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.008893    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3ee7a92b-openshift-default-dockercfg-mcs82\""
Sep 19 10:50:33.015589 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.015560    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"]
Sep 19 10:50:33.031409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.031374    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:33.191974 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.191936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf758\" (UniqueName: \"kubernetes.io/projected/15ff81ed-6770-4e92-9201-1e36f692392c-kube-api-access-pf758\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.191974 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.191975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ff81ed-6770-4e92-9201-1e36f692392c-istio-token\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.192213 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.192002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-credential-socket\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.192213 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.192020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ff81ed-6770-4e92-9201-1e36f692392c-istiod-ca-cert\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.192213 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.192088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-istio-envoy\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.192213 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.192126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-workload-certs\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.192213 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.192164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-istio-data\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.192213 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.192199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ff81ed-6770-4e92-9201-1e36f692392c-istio-podinfo\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.192462 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.192269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-workload-socket\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.293698 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.293587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-workload-certs\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.293698 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.293656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-istio-data\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.293991 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.293705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ff81ed-6770-4e92-9201-1e36f692392c-istio-podinfo\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.293991 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.293739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-workload-socket\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.293991 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.293830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf758\" (UniqueName: \"kubernetes.io/projected/15ff81ed-6770-4e92-9201-1e36f692392c-kube-api-access-pf758\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.293991 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.293856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ff81ed-6770-4e92-9201-1e36f692392c-istio-token\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.293991 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.293888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-credential-socket\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.293991 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.293921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ff81ed-6770-4e92-9201-1e36f692392c-istiod-ca-cert\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.293991 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.293970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-istio-envoy\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.294350 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.294007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-workload-certs\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.294350 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.294058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-istio-data\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.294350 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.294293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-credential-socket\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.294459 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.294405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-workload-socket\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.294693 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.294669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ff81ed-6770-4e92-9201-1e36f692392c-istiod-ca-cert\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.296450 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.296420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-istio-envoy\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.296539 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.296486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ff81ed-6770-4e92-9201-1e36f692392c-istio-podinfo\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.301808 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.301575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf758\" (UniqueName: \"kubernetes.io/projected/15ff81ed-6770-4e92-9201-1e36f692392c-kube-api-access-pf758\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.301808 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.301776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ff81ed-6770-4e92-9201-1e36f692392c-istio-token\") pod \"e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") " pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.317846 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.317804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:33.480265 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.480230    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:33.596843 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.596817    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"]
Sep 19 10:50:33.599194 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:50:33.599171    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ff81ed_6770_4e92_9201_1e36f692392c.slice/crio-ca9612ef01eaa744564ba3d9fac6e02cdaf36f9f2e8a1868860dd9467e022aa0 WatchSource:0}: Error finding container ca9612ef01eaa744564ba3d9fac6e02cdaf36f9f2e8a1868860dd9467e022aa0: Status 404 returned error can't find the container with id ca9612ef01eaa744564ba3d9fac6e02cdaf36f9f2e8a1868860dd9467e022aa0
Sep 19 10:50:33.601874 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.601836    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:50:33.601967 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.601908    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:50:33.601967 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.601943    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:50:33.650297 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.650244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:33.857576 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.857480    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f914ab-e8c7-4e96-922a-b6a838980465" path="/var/lib/kubelet/pods/75f914ab-e8c7-4e96-922a-b6a838980465/volumes"
Sep 19 10:50:33.857965 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.857947    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beda4ff5-3060-4862-8a90-b54b8806f09d" path="/var/lib/kubelet/pods/beda4ff5-3060-4862-8a90-b54b8806f09d/volumes"
Sep 19 10:50:33.889472 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.889436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" event={"ID":"15ff81ed-6770-4e92-9201-1e36f692392c","Type":"ContainerStarted","Data":"1b46da9f8bf941f7991b9c6a903aedcb8d28b4ee765651d958a4345ea3724a39"}
Sep 19 10:50:33.889688 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.889498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" event={"ID":"15ff81ed-6770-4e92-9201-1e36f692392c","Type":"ContainerStarted","Data":"ca9612ef01eaa744564ba3d9fac6e02cdaf36f9f2e8a1868860dd9467e022aa0"}
Sep 19 10:50:33.910927 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:33.910861    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" podStartSLOduration=1.9108444489999998 podStartE2EDuration="1.910844449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:50:33.908809547 +0000 UTC m=+878.567554708" watchObservedRunningTime="2026-09-19 10:50:33.910844449 +0000 UTC m=+878.569589610"
Sep 19 10:50:34.318849 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:34.318808    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:34.324324 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:34.324294    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:34.624409 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:34.624304    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:34.894829 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:34.894702    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:34.895723 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:34.895703    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:50:35.300900 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:35.300860    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:35.401288 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:35.401239    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:35.504430 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:35.504395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:36.164934 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:36.164897    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:36.922659 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:36.922608    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:37.518435 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.518406    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:50:37.536809 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.536703    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-credential-socket\") pod \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") "
Sep 19 10:50:37.536809 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.536743    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-workload-certs\") pod \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") "
Sep 19 10:50:37.536809 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.536799    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89pw8\" (UniqueName: \"kubernetes.io/projected/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-kube-api-access-89pw8\") pod \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") "
Sep 19 10:50:37.537318 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.536835    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istiod-ca-cert\") pod \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") "
Sep 19 10:50:37.537318 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.536857    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-token\") pod \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") "
Sep 19 10:50:37.537318 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.536897    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-envoy\") pod \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") "
Sep 19 10:50:37.537318 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.536943    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-podinfo\") pod \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") "
Sep 19 10:50:37.537318 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.537010    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-workload-socket\") pod \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") "
Sep 19 10:50:37.537318 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.537013    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "247f7a92-6a26-4f8e-9e71-19ce08ab7b76" (UID: "247f7a92-6a26-4f8e-9e71-19ce08ab7b76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:50:37.537318 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.537115    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "247f7a92-6a26-4f8e-9e71-19ce08ab7b76" (UID: "247f7a92-6a26-4f8e-9e71-19ce08ab7b76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:50:37.537790 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.537396    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "247f7a92-6a26-4f8e-9e71-19ce08ab7b76" (UID: "247f7a92-6a26-4f8e-9e71-19ce08ab7b76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:50:37.537790 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.537435    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "247f7a92-6a26-4f8e-9e71-19ce08ab7b76" (UID: "247f7a92-6a26-4f8e-9e71-19ce08ab7b76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:50:37.537790 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.537538    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-data\") pod \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\" (UID: \"247f7a92-6a26-4f8e-9e71-19ce08ab7b76\") "
Sep 19 10:50:37.538352 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.537802    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-data" (OuterVolumeSpecName: "istio-data") pod "247f7a92-6a26-4f8e-9e71-19ce08ab7b76" (UID: "247f7a92-6a26-4f8e-9e71-19ce08ab7b76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:50:37.538352 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.538308    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istiod-ca-cert\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:37.538352 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.538330    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-workload-socket\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:37.538514 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.538361    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-data\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:37.538514 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.538375    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-credential-socket\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:37.538514 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.538388    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-workload-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:37.540459 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.540432    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "247f7a92-6a26-4f8e-9e71-19ce08ab7b76" (UID: "247f7a92-6a26-4f8e-9e71-19ce08ab7b76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 10:50:37.540614 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.540581    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-kube-api-access-89pw8" (OuterVolumeSpecName: "kube-api-access-89pw8") pod "247f7a92-6a26-4f8e-9e71-19ce08ab7b76" (UID: "247f7a92-6a26-4f8e-9e71-19ce08ab7b76"). InnerVolumeSpecName "kube-api-access-89pw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:50:37.541113 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.541066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-token" (OuterVolumeSpecName: "istio-token") pod "247f7a92-6a26-4f8e-9e71-19ce08ab7b76" (UID: "247f7a92-6a26-4f8e-9e71-19ce08ab7b76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:50:37.541245 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.541203    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "247f7a92-6a26-4f8e-9e71-19ce08ab7b76" (UID: "247f7a92-6a26-4f8e-9e71-19ce08ab7b76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:50:37.557138 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.557094    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:37.639135 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.639102    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89pw8\" (UniqueName: \"kubernetes.io/projected/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-kube-api-access-89pw8\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:37.639135 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.639131    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-token\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:37.639135 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.639142    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-envoy\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:37.639387 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.639150    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247f7a92-6a26-4f8e-9e71-19ce08ab7b76-istio-podinfo\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:50:37.912227 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.912190    2580 generic.go:356] "Generic (PLEG): container finished" podID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerID="77dc45001afbb089be35557ecef2c948dc22b6a034e14ea7c0d9cd74c68ff7f5" exitCode=0
Sep 19 10:50:37.912431 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.912261    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"
Sep 19 10:50:37.912431 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.912279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" event={"ID":"247f7a92-6a26-4f8e-9e71-19ce08ab7b76","Type":"ContainerDied","Data":"77dc45001afbb089be35557ecef2c948dc22b6a034e14ea7c0d9cd74c68ff7f5"}
Sep 19 10:50:37.912431 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.912361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv" event={"ID":"247f7a92-6a26-4f8e-9e71-19ce08ab7b76","Type":"ContainerDied","Data":"88241760168d0213bccce66522c798f3d592b1a8789eddb4abd2b74665dd3f1c"}
Sep 19 10:50:37.912431 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.912384    2580 scope.go:117] "RemoveContainer" containerID="77dc45001afbb089be35557ecef2c948dc22b6a034e14ea7c0d9cd74c68ff7f5"
Sep 19 10:50:37.921462 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.921437    2580 scope.go:117] "RemoveContainer" containerID="4c7318add840cc43540fd0a445bda05759498c58998266cf5aee75831d29367e"
Sep 19 10:50:37.927615 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.927587    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"]
Sep 19 10:50:37.930535 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.930513    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d303059f-openshift-default-6c849d8b64-lwlqv"]
Sep 19 10:50:37.930872 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.930854    2580 scope.go:117] "RemoveContainer" containerID="77dc45001afbb089be35557ecef2c948dc22b6a034e14ea7c0d9cd74c68ff7f5"
Sep 19 10:50:37.931220 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:50:37.931200    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77dc45001afbb089be35557ecef2c948dc22b6a034e14ea7c0d9cd74c68ff7f5\": container with ID starting with 77dc45001afbb089be35557ecef2c948dc22b6a034e14ea7c0d9cd74c68ff7f5 not found: ID does not exist" containerID="77dc45001afbb089be35557ecef2c948dc22b6a034e14ea7c0d9cd74c68ff7f5"
Sep 19 10:50:37.931282 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.931229    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77dc45001afbb089be35557ecef2c948dc22b6a034e14ea7c0d9cd74c68ff7f5"} err="failed to get container status \"77dc45001afbb089be35557ecef2c948dc22b6a034e14ea7c0d9cd74c68ff7f5\": rpc error: code = NotFound desc = could not find container \"77dc45001afbb089be35557ecef2c948dc22b6a034e14ea7c0d9cd74c68ff7f5\": container with ID starting with 77dc45001afbb089be35557ecef2c948dc22b6a034e14ea7c0d9cd74c68ff7f5 not found: ID does not exist"
Sep 19 10:50:37.931282 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.931251    2580 scope.go:117] "RemoveContainer" containerID="4c7318add840cc43540fd0a445bda05759498c58998266cf5aee75831d29367e"
Sep 19 10:50:37.931522 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:50:37.931505    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7318add840cc43540fd0a445bda05759498c58998266cf5aee75831d29367e\": container with ID starting with 4c7318add840cc43540fd0a445bda05759498c58998266cf5aee75831d29367e not found: ID does not exist" containerID="4c7318add840cc43540fd0a445bda05759498c58998266cf5aee75831d29367e"
Sep 19 10:50:37.931584 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:37.931531    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7318add840cc43540fd0a445bda05759498c58998266cf5aee75831d29367e"} err="failed to get container status \"4c7318add840cc43540fd0a445bda05759498c58998266cf5aee75831d29367e\": rpc error: code = NotFound desc = could not find container \"4c7318add840cc43540fd0a445bda05759498c58998266cf5aee75831d29367e\": container with ID starting with 4c7318add840cc43540fd0a445bda05759498c58998266cf5aee75831d29367e not found: ID does not exist"
Sep 19 10:50:38.309188 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:38.309154    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:38.523082 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:38.523044    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:50:38.990614 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:38.990582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:39.521447 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:39.521405    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:39.521650 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:39.521459    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:39.534297 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:39.534267    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:39.640069 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:39.640033    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:39.695515 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:39.695470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:39.856918 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:39.856835    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" path="/var/lib/kubelet/pods/247f7a92-6a26-4f8e-9e71-19ce08ab7b76/volumes"
Sep 19 10:50:39.923119 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:39.923082    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:39.935899 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:39.935869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 10:50:40.099384 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.099348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:40.198285 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.198246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:50:40.223862 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.223826    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"]
Sep 19 10:50:40.224238 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.224225    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerName="istio-proxy"
Sep 19 10:50:40.224297 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.224240    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerName="istio-proxy"
Sep 19 10:50:40.224330 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.224306    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerName="istio-proxy"
Sep 19 10:50:40.228816 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.228801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"
Sep 19 10:50:40.231070 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.231029    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3ee7a92b\""
Sep 19 10:50:40.235730 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.235702    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"]
Sep 19 10:50:40.266148 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.266109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e097826-d2b2-4e41-b8a1-aa62528a2766-plugins-config-volume\") pod \"payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz\" (UID: \"2e097826-d2b2-4e41-b8a1-aa62528a2766\") " pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"
Sep 19 10:50:40.266321 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.266254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2wqk\" (UniqueName: \"kubernetes.io/projected/2e097826-d2b2-4e41-b8a1-aa62528a2766-kube-api-access-p2wqk\") pod \"payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz\" (UID: \"2e097826-d2b2-4e41-b8a1-aa62528a2766\") " pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"
Sep 19 10:50:40.289319 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.289289    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"]
Sep 19 10:50:40.289732 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.289718    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerName="istio-proxy"
Sep 19 10:50:40.289784 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.289738    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerName="istio-proxy"
Sep 19 10:50:40.289893 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.289882    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="247f7a92-6a26-4f8e-9e71-19ce08ab7b76" containerName="istio-proxy"
Sep 19 10:50:40.292978 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.292921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"
Sep 19 10:50:40.295044 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.295021    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3ee7a92b-dockercfg-52x5w\""
Sep 19 10:50:40.299670 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.299646    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"]
Sep 19 10:50:40.367193 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.367150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e097826-d2b2-4e41-b8a1-aa62528a2766-plugins-config-volume\") pod \"payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz\" (UID: \"2e097826-d2b2-4e41-b8a1-aa62528a2766\") " pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"
Sep 19 10:50:40.367367 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.367229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4b2ffa-614d-44e4-a04e-fa3264fe2312-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb\" (UID: \"6f4b2ffa-614d-44e4-a04e-fa3264fe2312\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"
Sep 19 10:50:40.367367 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.367303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2wqk\" (UniqueName: \"kubernetes.io/projected/2e097826-d2b2-4e41-b8a1-aa62528a2766-kube-api-access-p2wqk\") pod \"payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz\" (UID: \"2e097826-d2b2-4e41-b8a1-aa62528a2766\") " pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"
Sep 19 10:50:40.367518 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.367422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p62gs\" (UniqueName: \"kubernetes.io/projected/6f4b2ffa-614d-44e4-a04e-fa3264fe2312-kube-api-access-p62gs\") pod \"payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb\" (UID: \"6f4b2ffa-614d-44e4-a04e-fa3264fe2312\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"
Sep 19 10:50:40.368190 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.368167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e097826-d2b2-4e41-b8a1-aa62528a2766-plugins-config-volume\") pod \"payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz\" (UID: \"2e097826-d2b2-4e41-b8a1-aa62528a2766\") " pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"
Sep 19 10:50:40.374676 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.374655    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2wqk\" (UniqueName: \"kubernetes.io/projected/2e097826-d2b2-4e41-b8a1-aa62528a2766-kube-api-access-p2wqk\") pod \"payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz\" (UID: \"2e097826-d2b2-4e41-b8a1-aa62528a2766\") " pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"
Sep 19 10:50:40.468836 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.468698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4b2ffa-614d-44e4-a04e-fa3264fe2312-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb\" (UID: \"6f4b2ffa-614d-44e4-a04e-fa3264fe2312\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"
Sep 19 10:50:40.469007 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.468847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p62gs\" (UniqueName: \"kubernetes.io/projected/6f4b2ffa-614d-44e4-a04e-fa3264fe2312-kube-api-access-p62gs\") pod \"payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb\" (UID: \"6f4b2ffa-614d-44e4-a04e-fa3264fe2312\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"
Sep 19 10:50:40.469647 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.469621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4b2ffa-614d-44e4-a04e-fa3264fe2312-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb\" (UID: \"6f4b2ffa-614d-44e4-a04e-fa3264fe2312\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"
Sep 19 10:50:40.476164 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.476133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p62gs\" (UniqueName: \"kubernetes.io/projected/6f4b2ffa-614d-44e4-a04e-fa3264fe2312-kube-api-access-p62gs\") pod \"payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb\" (UID: \"6f4b2ffa-614d-44e4-a04e-fa3264fe2312\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"
Sep 19 10:50:40.559402 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.559363    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"
Sep 19 10:50:40.614872 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.614836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"
Sep 19 10:50:40.834910 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.834718    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"]
Sep 19 10:50:40.837180 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:50:40.837153    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e097826_d2b2_4e41_b8a1_aa62528a2766.slice/crio-014757dd65d198832db1be0819058872af29413fa997146674fc7af9e75316d0 WatchSource:0}: Error finding container 014757dd65d198832db1be0819058872af29413fa997146674fc7af9e75316d0: Status 404 returned error can't find the container with id 014757dd65d198832db1be0819058872af29413fa997146674fc7af9e75316d0
Sep 19 10:50:40.895667 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.895643    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"]
Sep 19 10:50:40.899716 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:50:40.899682    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4b2ffa_614d_44e4_a04e_fa3264fe2312.slice/crio-4def11cc294c78f7136c37c54c3fe1f11bcc5b6906606ecb68fe75038841c314 WatchSource:0}: Error finding container 4def11cc294c78f7136c37c54c3fe1f11bcc5b6906606ecb68fe75038841c314: Status 404 returned error can't find the container with id 4def11cc294c78f7136c37c54c3fe1f11bcc5b6906606ecb68fe75038841c314
Sep 19 10:50:40.930094 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.930057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb" event={"ID":"6f4b2ffa-614d-44e4-a04e-fa3264fe2312","Type":"ContainerStarted","Data":"4def11cc294c78f7136c37c54c3fe1f11bcc5b6906606ecb68fe75038841c314"}
Sep 19 10:50:40.931825 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.931795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz" event={"ID":"2e097826-d2b2-4e41-b8a1-aa62528a2766","Type":"ContainerStarted","Data":"e97ef29cb03d817beeb4f738b62928a4e34174c5e8cf592e460e293fad535e98"}
Sep 19 10:50:40.931965 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.931846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz" event={"ID":"2e097826-d2b2-4e41-b8a1-aa62528a2766","Type":"ContainerStarted","Data":"014757dd65d198832db1be0819058872af29413fa997146674fc7af9e75316d0"}
Sep 19 10:50:40.947521 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:40.947474    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz" podStartSLOduration=0.947457524 podStartE2EDuration="947.457524ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:50:40.945852222 +0000 UTC m=+885.604597381" watchObservedRunningTime="2026-09-19 10:50:40.947457524 +0000 UTC m=+885.606202661"
Sep 19 10:50:41.938305 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:41.938260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb" event={"ID":"6f4b2ffa-614d-44e4-a04e-fa3264fe2312","Type":"ContainerStarted","Data":"3646ec52168371eb7d6ebb8dd554577714636890810e3b80cb860faf47863ab6"}
Sep 19 10:50:41.938748 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:41.938498    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"
Sep 19 10:50:41.938748 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:41.938549    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"
Sep 19 10:50:41.953848 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:41.953801    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb" podStartSLOduration=1.953786628 podStartE2EDuration="1.953786628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:50:41.952149876 +0000 UTC m=+886.610895036" watchObservedRunningTime="2026-09-19 10:50:41.953786628 +0000 UTC m=+886.612531787"
Sep 19 10:50:52.945688 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:52.945598    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"
Sep 19 10:50:52.945688 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:52.945668    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"
Sep 19 10:50:55.823030 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:55.822999    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 10:50:55.829161 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:55.829136    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 10:50:58.314399 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:50:58.314363    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:00.023661 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:00.023625    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:51:00.037216 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:00.037187    2580 generic.go:356] "Generic (PLEG): container finished" podID="15ff81ed-6770-4e92-9201-1e36f692392c" containerID="1b46da9f8bf941f7991b9c6a903aedcb8d28b4ee765651d958a4345ea3724a39" exitCode=0
Sep 19 10:51:00.037367 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:00.037259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" event={"ID":"15ff81ed-6770-4e92-9201-1e36f692392c","Type":"ContainerDied","Data":"1b46da9f8bf941f7991b9c6a903aedcb8d28b4ee765651d958a4345ea3724a39"}
Sep 19 10:51:00.037672 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:00.037654    2580 scope.go:117] "RemoveContainer" containerID="1b46da9f8bf941f7991b9c6a903aedcb8d28b4ee765651d958a4345ea3724a39"
Sep 19 10:51:00.038813 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:00.038788    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:51:00.038879 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:00.038859    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:51:00.038927 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:00.038892    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 10:51:01.043990 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:01.043949    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" event={"ID":"15ff81ed-6770-4e92-9201-1e36f692392c","Type":"ContainerStarted","Data":"c7b7c68cbf0c78fb62e335dd518bf47f972739ea6b64fc7a3ebf41f525d4fac3"}
Sep 19 10:51:01.318307 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:01.318216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:51:01.319448 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:01.319411    2580 patch_prober.go:28] interesting pod/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 19 10:51:01.319582 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:01.319498    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 19 10:51:01.414400 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:01.414365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:02.127670 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:02.127634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:02.319190 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:02.319154    2580 patch_prober.go:28] interesting pod/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 19 10:51:02.319381 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:02.319219    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 19 10:51:02.731707 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:02.731664    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:03.318393 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:03.318353    2580 patch_prober.go:28] interesting pod/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 19 10:51:03.318839 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:03.318413    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 19 10:51:03.347587 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:03.347543    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:03.983020 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:03.982976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:04.318736 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:04.318641    2580 patch_prober.go:28] interesting pod/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 19 10:51:04.318736 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:04.318715    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 19 10:51:04.563745 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:04.563700    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:04.895469 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:04.895421    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:51:05.318870 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:05.318827    2580 patch_prober.go:28] interesting pod/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 19 10:51:05.319285 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:05.318897    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 19 10:51:05.428794 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:05.428741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:06.066772 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:06.066719    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:06.341360 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:06.341265    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:51:06.344770 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:06.344726    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:51:06.714084 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:06.714053    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:07.416001 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:07.415965    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:08.094613 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.094579    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:08.603360 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.603327    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"]
Sep 19 10:51:08.606789 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.606768    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.609014 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.608993    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7bcaa9\"/\"kube-root-ca.crt\""
Sep 19 10:51:08.609706 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.609682    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7bcaa9\"/\"default-dockercfg-rtvzp\""
Sep 19 10:51:08.609861 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.609721    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7bcaa9\"/\"openshift-service-ca.crt\""
Sep 19 10:51:08.609861 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.609742    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7bcaa9\"/\"autores-model-7bcaa9-kserve-self-signed-certs\""
Sep 19 10:51:08.615141 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.615118    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"]
Sep 19 10:51:08.700539 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.700508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:08.751496 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.751451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-dshm\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.751695 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.751511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-tls-certs\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.751695 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.751582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5w22\" (UniqueName: \"kubernetes.io/projected/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-kube-api-access-j5w22\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.751695 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.751607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-tmp-dir\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.751695 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.751649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-model-cache\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.751882 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.751735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-home\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.833437 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.833398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:08.853131 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.853098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-dshm\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.853336 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.853157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-tls-certs\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.853336 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.853209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5w22\" (UniqueName: \"kubernetes.io/projected/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-kube-api-access-j5w22\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.853336 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.853243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-tmp-dir\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.853336 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.853293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-model-cache\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.853336 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.853328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-home\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.853708 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.853629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-tmp-dir\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.853708 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.853672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-model-cache\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.853895 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.853724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-home\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.855634 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.855617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-dshm\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.855810 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.855794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-tls-certs\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.860656 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.860625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5w22\" (UniqueName: \"kubernetes.io/projected/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-kube-api-access-j5w22\") pod \"autores-model-7bcaa9-kserve-87444df95-4c7sv\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") " pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:08.917587 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:08.917530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:09.205271 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:09.205243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"]
Sep 19 10:51:09.207439 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:51:09.207410    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda31e39c7_37aa_451f_bd3b_f68bdbc341e5.slice/crio-93976e899f893a3366b75189631e339e0ae49683401ae62da2b4e3c233d74c84 WatchSource:0}: Error finding container 93976e899f893a3366b75189631e339e0ae49683401ae62da2b4e3c233d74c84: Status 404 returned error can't find the container with id 93976e899f893a3366b75189631e339e0ae49683401ae62da2b4e3c233d74c84
Sep 19 10:51:09.562256 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:09.562144    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:10.101400 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:10.101364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv" event={"ID":"a31e39c7-37aa-451f-bd3b-f68bdbc341e5","Type":"ContainerStarted","Data":"ecb553ff8872046121fd21eaab3420d79281ef804d8c168de55776d52721fd84"}
Sep 19 10:51:10.101815 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:10.101413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv" event={"ID":"a31e39c7-37aa-451f-bd3b-f68bdbc341e5","Type":"ContainerStarted","Data":"93976e899f893a3366b75189631e339e0ae49683401ae62da2b4e3c233d74c84"}
Sep 19 10:51:10.118143 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:10.118093    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv" podStartSLOduration=2.118077431 podStartE2EDuration="2.118077431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:51:10.116122673 +0000 UTC m=+914.774867833" watchObservedRunningTime="2026-09-19 10:51:10.118077431 +0000 UTC m=+914.776822590"
Sep 19 10:51:10.379070 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:10.378981    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:14.812518 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:14.812477    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:51:16.990069 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:16.989954    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:17.007009 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.006973    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"]
Sep 19 10:51:17.007200 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.007179    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz" podUID="2e097826-d2b2-4e41-b8a1-aa62528a2766" containerName="payload-processing" containerID="cri-o://e97ef29cb03d817beeb4f738b62928a4e34174c5e8cf592e460e293fad535e98" gracePeriod=30
Sep 19 10:51:17.014401 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.014347    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"]
Sep 19 10:51:17.014638 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.014588    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb" podUID="6f4b2ffa-614d-44e4-a04e-fa3264fe2312" containerName="payload-pre-processing" containerID="cri-o://3646ec52168371eb7d6ebb8dd554577714636890810e3b80cb860faf47863ab6" gracePeriod=30
Sep 19 10:51:17.089799 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.089739    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:17.140670 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.140639    2580 generic.go:356] "Generic (PLEG): container finished" podID="6f4b2ffa-614d-44e4-a04e-fa3264fe2312" containerID="3646ec52168371eb7d6ebb8dd554577714636890810e3b80cb860faf47863ab6" exitCode=0
Sep 19 10:51:17.140842 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.140700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb" event={"ID":"6f4b2ffa-614d-44e4-a04e-fa3264fe2312","Type":"ContainerDied","Data":"3646ec52168371eb7d6ebb8dd554577714636890810e3b80cb860faf47863ab6"}
Sep 19 10:51:17.142588 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.142563    2580 generic.go:356] "Generic (PLEG): container finished" podID="2e097826-d2b2-4e41-b8a1-aa62528a2766" containerID="e97ef29cb03d817beeb4f738b62928a4e34174c5e8cf592e460e293fad535e98" exitCode=0
Sep 19 10:51:17.142704 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.142614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz" event={"ID":"2e097826-d2b2-4e41-b8a1-aa62528a2766","Type":"ContainerDied","Data":"e97ef29cb03d817beeb4f738b62928a4e34174c5e8cf592e460e293fad535e98"}
Sep 19 10:51:17.169586 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.169550    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:17.226315 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.226284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:17.286076 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.286042    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"
Sep 19 10:51:17.293405 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.293363    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"]
Sep 19 10:51:17.294103 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.294065    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" containerName="istio-proxy" containerID="cri-o://c7b7c68cbf0c78fb62e335dd518bf47f972739ea6b64fc7a3ebf41f525d4fac3" gracePeriod=30
Sep 19 10:51:17.310499 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.310466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:17.324638 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.324610    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"
Sep 19 10:51:17.432793 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.432714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2wqk\" (UniqueName: \"kubernetes.io/projected/2e097826-d2b2-4e41-b8a1-aa62528a2766-kube-api-access-p2wqk\") pod \"2e097826-d2b2-4e41-b8a1-aa62528a2766\" (UID: \"2e097826-d2b2-4e41-b8a1-aa62528a2766\") "
Sep 19 10:51:17.432981 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.432821    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p62gs\" (UniqueName: \"kubernetes.io/projected/6f4b2ffa-614d-44e4-a04e-fa3264fe2312-kube-api-access-p62gs\") pod \"6f4b2ffa-614d-44e4-a04e-fa3264fe2312\" (UID: \"6f4b2ffa-614d-44e4-a04e-fa3264fe2312\") "
Sep 19 10:51:17.432981 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.432865    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e097826-d2b2-4e41-b8a1-aa62528a2766-plugins-config-volume\") pod \"2e097826-d2b2-4e41-b8a1-aa62528a2766\" (UID: \"2e097826-d2b2-4e41-b8a1-aa62528a2766\") "
Sep 19 10:51:17.432981 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.432882    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4b2ffa-614d-44e4-a04e-fa3264fe2312-plugins-config-volume\") pod \"6f4b2ffa-614d-44e4-a04e-fa3264fe2312\" (UID: \"6f4b2ffa-614d-44e4-a04e-fa3264fe2312\") "
Sep 19 10:51:17.433280 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.433245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e097826-d2b2-4e41-b8a1-aa62528a2766-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e097826-d2b2-4e41-b8a1-aa62528a2766" (UID: "2e097826-d2b2-4e41-b8a1-aa62528a2766"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:51:17.433280 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.433271    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4b2ffa-614d-44e4-a04e-fa3264fe2312-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f4b2ffa-614d-44e4-a04e-fa3264fe2312" (UID: "6f4b2ffa-614d-44e4-a04e-fa3264fe2312"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:51:17.435100 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.435075    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e097826-d2b2-4e41-b8a1-aa62528a2766-kube-api-access-p2wqk" (OuterVolumeSpecName: "kube-api-access-p2wqk") pod "2e097826-d2b2-4e41-b8a1-aa62528a2766" (UID: "2e097826-d2b2-4e41-b8a1-aa62528a2766"). InnerVolumeSpecName "kube-api-access-p2wqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:51:17.435195 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.435112    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4b2ffa-614d-44e4-a04e-fa3264fe2312-kube-api-access-p62gs" (OuterVolumeSpecName: "kube-api-access-p62gs") pod "6f4b2ffa-614d-44e4-a04e-fa3264fe2312" (UID: "6f4b2ffa-614d-44e4-a04e-fa3264fe2312"). InnerVolumeSpecName "kube-api-access-p62gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:51:17.534174 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.534084    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2wqk\" (UniqueName: \"kubernetes.io/projected/2e097826-d2b2-4e41-b8a1-aa62528a2766-kube-api-access-p2wqk\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:51:17.534174 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.534114    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p62gs\" (UniqueName: \"kubernetes.io/projected/6f4b2ffa-614d-44e4-a04e-fa3264fe2312-kube-api-access-p62gs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:51:17.534174 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.534123    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e097826-d2b2-4e41-b8a1-aa62528a2766-plugins-config-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:51:17.534174 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.534131    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4b2ffa-614d-44e4-a04e-fa3264fe2312-plugins-config-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:51:17.612451 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.612413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:17.959786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:17.959721    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:18.059436 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.059387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:18.149126 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.149039    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"
Sep 19 10:51:18.149126 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.149080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz" event={"ID":"2e097826-d2b2-4e41-b8a1-aa62528a2766","Type":"ContainerDied","Data":"014757dd65d198832db1be0819058872af29413fa997146674fc7af9e75316d0"}
Sep 19 10:51:18.149357 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.149146    2580 scope.go:117] "RemoveContainer" containerID="e97ef29cb03d817beeb4f738b62928a4e34174c5e8cf592e460e293fad535e98"
Sep 19 10:51:18.150629 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.150603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb" event={"ID":"6f4b2ffa-614d-44e4-a04e-fa3264fe2312","Type":"ContainerDied","Data":"4def11cc294c78f7136c37c54c3fe1f11bcc5b6906606ecb68fe75038841c314"}
Sep 19 10:51:18.150746 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.150645    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"
Sep 19 10:51:18.158118 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.158097    2580 scope.go:117] "RemoveContainer" containerID="3646ec52168371eb7d6ebb8dd554577714636890810e3b80cb860faf47863ab6"
Sep 19 10:51:18.161736 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.161709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:18.169866 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.169844    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"]
Sep 19 10:51:18.174116 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.174091    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3ee7a92b-598f85f89d-bd7cb"]
Sep 19 10:51:18.188426 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.188388    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"]
Sep 19 10:51:18.188616 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.188441    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3ee7a92b-5d4dd44db5-bhfpz"]
Sep 19 10:51:18.593308 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.593274    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:18.917938 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.917843    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:18.917938 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.917889    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:18.930285 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:18.930255    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:19.171152 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:19.171060    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:51:19.233378 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:19.233337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:19.594134 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:19.594097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:19.857054 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:19.856976    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e097826-d2b2-4e41-b8a1-aa62528a2766" path="/var/lib/kubelet/pods/2e097826-d2b2-4e41-b8a1-aa62528a2766/volumes"
Sep 19 10:51:19.857330 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:19.857318    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f4b2ffa-614d-44e4-a04e-fa3264fe2312" path="/var/lib/kubelet/pods/6f4b2ffa-614d-44e4-a04e-fa3264fe2312/volumes"
Sep 19 10:51:19.900636 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:19.900604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:20.535709 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:20.535674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:21.154783 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:21.154720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:21.344792 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:21.344725    2580 patch_prober.go:28] interesting pod/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 19 10:51:21.344976 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:21.344813    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 10:51:22.560837 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.560814    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:51:22.582167 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582080    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-workload-certs\") pod \"15ff81ed-6770-4e92-9201-1e36f692392c\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") "
Sep 19 10:51:22.582167 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582148    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-istio-envoy\") pod \"15ff81ed-6770-4e92-9201-1e36f692392c\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") "
Sep 19 10:51:22.582379 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582178    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-istio-data\") pod \"15ff81ed-6770-4e92-9201-1e36f692392c\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") "
Sep 19 10:51:22.582379 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582211    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf758\" (UniqueName: \"kubernetes.io/projected/15ff81ed-6770-4e92-9201-1e36f692392c-kube-api-access-pf758\") pod \"15ff81ed-6770-4e92-9201-1e36f692392c\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") "
Sep 19 10:51:22.582379 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582238    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ff81ed-6770-4e92-9201-1e36f692392c-istio-token\") pod \"15ff81ed-6770-4e92-9201-1e36f692392c\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") "
Sep 19 10:51:22.582379 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582273    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-credential-socket\") pod \"15ff81ed-6770-4e92-9201-1e36f692392c\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") "
Sep 19 10:51:22.582379 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582300    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ff81ed-6770-4e92-9201-1e36f692392c-istio-podinfo\") pod \"15ff81ed-6770-4e92-9201-1e36f692392c\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") "
Sep 19 10:51:22.582379 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582334    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ff81ed-6770-4e92-9201-1e36f692392c-istiod-ca-cert\") pod \"15ff81ed-6770-4e92-9201-1e36f692392c\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") "
Sep 19 10:51:22.582379 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582373    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-workload-socket\") pod \"15ff81ed-6770-4e92-9201-1e36f692392c\" (UID: \"15ff81ed-6770-4e92-9201-1e36f692392c\") "
Sep 19 10:51:22.582782 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582416    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15ff81ed-6770-4e92-9201-1e36f692392c" (UID: "15ff81ed-6770-4e92-9201-1e36f692392c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:51:22.582782 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582727    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-workload-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:51:22.582889 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582813    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15ff81ed-6770-4e92-9201-1e36f692392c" (UID: "15ff81ed-6770-4e92-9201-1e36f692392c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:51:22.582889 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582820    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ff81ed-6770-4e92-9201-1e36f692392c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15ff81ed-6770-4e92-9201-1e36f692392c" (UID: "15ff81ed-6770-4e92-9201-1e36f692392c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:51:22.582982 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.582956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15ff81ed-6770-4e92-9201-1e36f692392c" (UID: "15ff81ed-6770-4e92-9201-1e36f692392c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:51:22.583499 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.583464    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-istio-data" (OuterVolumeSpecName: "istio-data") pod "15ff81ed-6770-4e92-9201-1e36f692392c" (UID: "15ff81ed-6770-4e92-9201-1e36f692392c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:51:22.585282 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.585241    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ff81ed-6770-4e92-9201-1e36f692392c-kube-api-access-pf758" (OuterVolumeSpecName: "kube-api-access-pf758") pod "15ff81ed-6770-4e92-9201-1e36f692392c" (UID: "15ff81ed-6770-4e92-9201-1e36f692392c"). InnerVolumeSpecName "kube-api-access-pf758". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:51:22.585494 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.585457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15ff81ed-6770-4e92-9201-1e36f692392c" (UID: "15ff81ed-6770-4e92-9201-1e36f692392c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:51:22.585698 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.585669    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ff81ed-6770-4e92-9201-1e36f692392c-istio-token" (OuterVolumeSpecName: "istio-token") pod "15ff81ed-6770-4e92-9201-1e36f692392c" (UID: "15ff81ed-6770-4e92-9201-1e36f692392c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:51:22.585852 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.585830    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15ff81ed-6770-4e92-9201-1e36f692392c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15ff81ed-6770-4e92-9201-1e36f692392c" (UID: "15ff81ed-6770-4e92-9201-1e36f692392c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 10:51:22.683341 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.683298    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-istio-envoy\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:51:22.683341 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.683333    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-istio-data\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:51:22.683341 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.683344    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf758\" (UniqueName: \"kubernetes.io/projected/15ff81ed-6770-4e92-9201-1e36f692392c-kube-api-access-pf758\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:51:22.683341 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.683354    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ff81ed-6770-4e92-9201-1e36f692392c-istio-token\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:51:22.683633 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.683364    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-credential-socket\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:51:22.683633 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.683372    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ff81ed-6770-4e92-9201-1e36f692392c-istio-podinfo\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:51:22.683633 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.683382    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ff81ed-6770-4e92-9201-1e36f692392c-istiod-ca-cert\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:51:22.683633 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:22.683391    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff81ed-6770-4e92-9201-1e36f692392c-workload-socket\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:51:23.180523 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.180486    2580 generic.go:356] "Generic (PLEG): container finished" podID="15ff81ed-6770-4e92-9201-1e36f692392c" containerID="c7b7c68cbf0c78fb62e335dd518bf47f972739ea6b64fc7a3ebf41f525d4fac3" exitCode=0
Sep 19 10:51:23.180710 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.180541    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"
Sep 19 10:51:23.180710 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.180571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" event={"ID":"15ff81ed-6770-4e92-9201-1e36f692392c","Type":"ContainerDied","Data":"c7b7c68cbf0c78fb62e335dd518bf47f972739ea6b64fc7a3ebf41f525d4fac3"}
Sep 19 10:51:23.180710 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.180628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n" event={"ID":"15ff81ed-6770-4e92-9201-1e36f692392c","Type":"ContainerDied","Data":"ca9612ef01eaa744564ba3d9fac6e02cdaf36f9f2e8a1868860dd9467e022aa0"}
Sep 19 10:51:23.180710 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.180652    2580 scope.go:117] "RemoveContainer" containerID="c7b7c68cbf0c78fb62e335dd518bf47f972739ea6b64fc7a3ebf41f525d4fac3"
Sep 19 10:51:23.190466 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.190442    2580 scope.go:117] "RemoveContainer" containerID="1b46da9f8bf941f7991b9c6a903aedcb8d28b4ee765651d958a4345ea3724a39"
Sep 19 10:51:23.199316 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.199296    2580 scope.go:117] "RemoveContainer" containerID="c7b7c68cbf0c78fb62e335dd518bf47f972739ea6b64fc7a3ebf41f525d4fac3"
Sep 19 10:51:23.199657 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:51:23.199635    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b7c68cbf0c78fb62e335dd518bf47f972739ea6b64fc7a3ebf41f525d4fac3\": container with ID starting with c7b7c68cbf0c78fb62e335dd518bf47f972739ea6b64fc7a3ebf41f525d4fac3 not found: ID does not exist" containerID="c7b7c68cbf0c78fb62e335dd518bf47f972739ea6b64fc7a3ebf41f525d4fac3"
Sep 19 10:51:23.199728 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.199663    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b7c68cbf0c78fb62e335dd518bf47f972739ea6b64fc7a3ebf41f525d4fac3"} err="failed to get container status \"c7b7c68cbf0c78fb62e335dd518bf47f972739ea6b64fc7a3ebf41f525d4fac3\": rpc error: code = NotFound desc = could not find container \"c7b7c68cbf0c78fb62e335dd518bf47f972739ea6b64fc7a3ebf41f525d4fac3\": container with ID starting with c7b7c68cbf0c78fb62e335dd518bf47f972739ea6b64fc7a3ebf41f525d4fac3 not found: ID does not exist"
Sep 19 10:51:23.199728 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.199683    2580 scope.go:117] "RemoveContainer" containerID="1b46da9f8bf941f7991b9c6a903aedcb8d28b4ee765651d958a4345ea3724a39"
Sep 19 10:51:23.200066 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:51:23.200043    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b46da9f8bf941f7991b9c6a903aedcb8d28b4ee765651d958a4345ea3724a39\": container with ID starting with 1b46da9f8bf941f7991b9c6a903aedcb8d28b4ee765651d958a4345ea3724a39 not found: ID does not exist" containerID="1b46da9f8bf941f7991b9c6a903aedcb8d28b4ee765651d958a4345ea3724a39"
Sep 19 10:51:23.200225 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.200074    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b46da9f8bf941f7991b9c6a903aedcb8d28b4ee765651d958a4345ea3724a39"} err="failed to get container status \"1b46da9f8bf941f7991b9c6a903aedcb8d28b4ee765651d958a4345ea3724a39\": rpc error: code = NotFound desc = could not find container \"1b46da9f8bf941f7991b9c6a903aedcb8d28b4ee765651d958a4345ea3724a39\": container with ID starting with 1b46da9f8bf941f7991b9c6a903aedcb8d28b4ee765651d958a4345ea3724a39 not found: ID does not exist"
Sep 19 10:51:23.202568 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.202542    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"]
Sep 19 10:51:23.206199 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.206177    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3ee7a92b-openshift-default-785c9f7f64-mv89n"]
Sep 19 10:51:23.608255 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.608224    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:23.658039 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.657993    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:23.856511 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:23.856469    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" path="/var/lib/kubelet/pods/15ff81ed-6770-4e92-9201-1e36f692392c/volumes"
Sep 19 10:51:25.226417 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.226376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:25.293445 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.293409    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:25.362420 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.362376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:25.466085 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.466047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:25.913183 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.913155    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"]
Sep 19 10:51:25.913580 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.913563    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" containerName="istio-proxy"
Sep 19 10:51:25.913671 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.913584    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" containerName="istio-proxy"
Sep 19 10:51:25.913671 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.913597    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e097826-d2b2-4e41-b8a1-aa62528a2766" containerName="payload-processing"
Sep 19 10:51:25.913671 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.913605    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e097826-d2b2-4e41-b8a1-aa62528a2766" containerName="payload-processing"
Sep 19 10:51:25.913671 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.913617    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4b2ffa-614d-44e4-a04e-fa3264fe2312" containerName="payload-pre-processing"
Sep 19 10:51:25.913671 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.913627    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4b2ffa-614d-44e4-a04e-fa3264fe2312" containerName="payload-pre-processing"
Sep 19 10:51:25.913671 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.913646    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" containerName="istio-proxy"
Sep 19 10:51:25.913671 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.913653    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" containerName="istio-proxy"
Sep 19 10:51:25.914048 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.913784    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e097826-d2b2-4e41-b8a1-aa62528a2766" containerName="payload-processing"
Sep 19 10:51:25.914048 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.913797    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" containerName="istio-proxy"
Sep 19 10:51:25.914048 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.913811    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4b2ffa-614d-44e4-a04e-fa3264fe2312" containerName="payload-pre-processing"
Sep 19 10:51:25.915769 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.915711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:25.917945 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.917929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:25.920913 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.920888    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6a0d36\"/\"default-dockercfg-dglbn\""
Sep 19 10:51:25.921026 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.920908    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6a0d36\"/\"openshift-service-ca.crt\""
Sep 19 10:51:25.921026 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.920912    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6a0d36\"/\"kube-root-ca.crt\""
Sep 19 10:51:25.921026 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.920919    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6a0d36\"/\"test-model-6a0d36-kserve-self-signed-certs\""
Sep 19 10:51:25.924772 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:25.924731    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"]
Sep 19 10:51:26.010322 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.010288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-model-cache\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.010322 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.010324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbx86\" (UniqueName: \"kubernetes.io/projected/dd962b8a-8a90-42ea-8d4b-9295601dafd8-kube-api-access-nbx86\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.010542 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.010343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-home\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.010542 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.010475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-tmp-dir\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.010646 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.010558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-dshm\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.010646 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.010583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd962b8a-8a90-42ea-8d4b-9295601dafd8-tls-certs\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.011982 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.011961    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:26.111022 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.110985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd962b8a-8a90-42ea-8d4b-9295601dafd8-tls-certs\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.111212 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.111033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-model-cache\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.111212 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.111061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbx86\" (UniqueName: \"kubernetes.io/projected/dd962b8a-8a90-42ea-8d4b-9295601dafd8-kube-api-access-nbx86\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.111212 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.111085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-home\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.111212 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.111144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-tmp-dir\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.111212 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.111210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-dshm\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.111520 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.111435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-model-cache\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.111588 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.111542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-home\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.111694 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.111675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-tmp-dir\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.113656 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.113631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-dshm\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.113774 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.113679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd962b8a-8a90-42ea-8d4b-9295601dafd8-tls-certs\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.119234 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.119209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbx86\" (UniqueName: \"kubernetes.io/projected/dd962b8a-8a90-42ea-8d4b-9295601dafd8-kube-api-access-nbx86\") pod \"test-model-6a0d36-kserve-77c7488969-5srcs\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") " pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.227784 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.227724    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:26.503024 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:26.502994    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"]
Sep 19 10:51:26.505156 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:51:26.505125    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd962b8a_8a90_42ea_8d4b_9295601dafd8.slice/crio-8067a2cce424edf1fad2c6ed3def978ae0251be1b78eccbbd9487372555b81ba WatchSource:0}: Error finding container 8067a2cce424edf1fad2c6ed3def978ae0251be1b78eccbbd9487372555b81ba: Status 404 returned error can't find the container with id 8067a2cce424edf1fad2c6ed3def978ae0251be1b78eccbbd9487372555b81ba
Sep 19 10:51:27.204538 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:27.204502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs" event={"ID":"dd962b8a-8a90-42ea-8d4b-9295601dafd8","Type":"ContainerStarted","Data":"eeeac578968125a1d02591300ecea7a95dd36db6ffa77d17b48f23a7d8cda301"}
Sep 19 10:51:27.204538 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:27.204543    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs" event={"ID":"dd962b8a-8a90-42ea-8d4b-9295601dafd8","Type":"ContainerStarted","Data":"8067a2cce424edf1fad2c6ed3def978ae0251be1b78eccbbd9487372555b81ba"}
Sep 19 10:51:27.219904 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:27.219857    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs" podStartSLOduration=2.219841174 podStartE2EDuration="2.219841174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:51:27.218775054 +0000 UTC m=+931.877520208" watchObservedRunningTime="2026-09-19 10:51:27.219841174 +0000 UTC m=+931.878586334"
Sep 19 10:51:28.262811 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:28.262770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:28.360341 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:28.360296    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:35.716026 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:35.715988    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:51:36.227919 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:36.227875    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:36.228108 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:36.227931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:36.240655 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:36.240630    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:36.265671 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:36.265636    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:51:43.727652 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:43.727611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:44.776090 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:44.776044    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:45.121016 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:45.120903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:45.820279 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:45.820242    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:46.446680 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:46.446634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:47.112678 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:47.112645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:47.739985 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:47.739945    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:48.802863 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:48.802828    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"]
Sep 19 10:51:48.822315 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:48.822277    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:51.776643 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:51.776600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:53.886885 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:53.886852    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:53.986336 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:53.986302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:54.080146 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:54.080107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:54.135004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:54.134967    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:54.427425 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:54.427389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:54.529592 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:54.529551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:55.140412 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:55.140375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:55.814501 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:55.814460    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:56.529986 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:56.529955    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:57.322181 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:57.322144    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:57.870936 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:57.870888    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:57.971051 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:57.971010    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:58.070954 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:58.070921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:51:58.912653 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:58.912617    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:51:59.509982 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:51:59.509950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:03.823785 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:03.823690    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv" podUID="a31e39c7-37aa-451f-bd3b-f68bdbc341e5" containerName="main" containerID="cri-o://ecb553ff8872046121fd21eaab3420d79281ef804d8c168de55776d52721fd84" gracePeriod=45
Sep 19 10:52:04.074572 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.074500    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:52:04.175312 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.175282    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-model-cache\") pod \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") "
Sep 19 10:52:04.175505 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.175358    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-tls-certs\") pod \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") "
Sep 19 10:52:04.175505 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.175379    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-dshm\") pod \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") "
Sep 19 10:52:04.175505 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.175396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5w22\" (UniqueName: \"kubernetes.io/projected/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-kube-api-access-j5w22\") pod \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") "
Sep 19 10:52:04.175505 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.175425    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-tmp-dir\") pod \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") "
Sep 19 10:52:04.175723 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.175526    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-home\") pod \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\" (UID: \"a31e39c7-37aa-451f-bd3b-f68bdbc341e5\") "
Sep 19 10:52:04.175723 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.175670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "a31e39c7-37aa-451f-bd3b-f68bdbc341e5" (UID: "a31e39c7-37aa-451f-bd3b-f68bdbc341e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:52:04.175855 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.175788    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a31e39c7-37aa-451f-bd3b-f68bdbc341e5" (UID: "a31e39c7-37aa-451f-bd3b-f68bdbc341e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:52:04.175855 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.175830    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-home" (OuterVolumeSpecName: "home") pod "a31e39c7-37aa-451f-bd3b-f68bdbc341e5" (UID: "a31e39c7-37aa-451f-bd3b-f68bdbc341e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:52:04.175974 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.175891    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:52:04.175974 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.175908    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:52:04.178049 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.178017    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-kube-api-access-j5w22" (OuterVolumeSpecName: "kube-api-access-j5w22") pod "a31e39c7-37aa-451f-bd3b-f68bdbc341e5" (UID: "a31e39c7-37aa-451f-bd3b-f68bdbc341e5"). InnerVolumeSpecName "kube-api-access-j5w22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:52:04.178049 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.178020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a31e39c7-37aa-451f-bd3b-f68bdbc341e5" (UID: "a31e39c7-37aa-451f-bd3b-f68bdbc341e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:52:04.178188 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.178094    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-dshm" (OuterVolumeSpecName: "dshm") pod "a31e39c7-37aa-451f-bd3b-f68bdbc341e5" (UID: "a31e39c7-37aa-451f-bd3b-f68bdbc341e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:52:04.276544 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.276510    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:52:04.276544 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.276537    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:52:04.276544 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.276546    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5w22\" (UniqueName: \"kubernetes.io/projected/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-kube-api-access-j5w22\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:52:04.276544 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.276556    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a31e39c7-37aa-451f-bd3b-f68bdbc341e5-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:52:04.400090 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.399996    2580 generic.go:356] "Generic (PLEG): container finished" podID="a31e39c7-37aa-451f-bd3b-f68bdbc341e5" containerID="ecb553ff8872046121fd21eaab3420d79281ef804d8c168de55776d52721fd84" exitCode=0
Sep 19 10:52:04.400090 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.400032    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"
Sep 19 10:52:04.400270 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.400088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv" event={"ID":"a31e39c7-37aa-451f-bd3b-f68bdbc341e5","Type":"ContainerDied","Data":"ecb553ff8872046121fd21eaab3420d79281ef804d8c168de55776d52721fd84"}
Sep 19 10:52:04.400270 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.400125    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv" event={"ID":"a31e39c7-37aa-451f-bd3b-f68bdbc341e5","Type":"ContainerDied","Data":"93976e899f893a3366b75189631e339e0ae49683401ae62da2b4e3c233d74c84"}
Sep 19 10:52:04.400270 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.400146    2580 scope.go:117] "RemoveContainer" containerID="ecb553ff8872046121fd21eaab3420d79281ef804d8c168de55776d52721fd84"
Sep 19 10:52:04.409520 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.409497    2580 scope.go:117] "RemoveContainer" containerID="ecb553ff8872046121fd21eaab3420d79281ef804d8c168de55776d52721fd84"
Sep 19 10:52:04.409860 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:52:04.409841    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb553ff8872046121fd21eaab3420d79281ef804d8c168de55776d52721fd84\": container with ID starting with ecb553ff8872046121fd21eaab3420d79281ef804d8c168de55776d52721fd84 not found: ID does not exist" containerID="ecb553ff8872046121fd21eaab3420d79281ef804d8c168de55776d52721fd84"
Sep 19 10:52:04.409920 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.409873    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb553ff8872046121fd21eaab3420d79281ef804d8c168de55776d52721fd84"} err="failed to get container status \"ecb553ff8872046121fd21eaab3420d79281ef804d8c168de55776d52721fd84\": rpc error: code = NotFound desc = could not find container \"ecb553ff8872046121fd21eaab3420d79281ef804d8c168de55776d52721fd84\": container with ID starting with ecb553ff8872046121fd21eaab3420d79281ef804d8c168de55776d52721fd84 not found: ID does not exist"
Sep 19 10:52:04.419719 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.419688    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"]
Sep 19 10:52:04.423463 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:04.423428    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-7bcaa9/autores-model-7bcaa9-kserve-87444df95-4c7sv"]
Sep 19 10:52:05.146576 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:05.146537    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:05.379866 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:05.379832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:05.816650 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:05.816615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:05.857362 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:05.857329    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a31e39c7-37aa-451f-bd3b-f68bdbc341e5" path="/var/lib/kubelet/pods/a31e39c7-37aa-451f-bd3b-f68bdbc341e5/volumes"
Sep 19 10:52:06.111805 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:06.111696    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:06.564681 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:06.564645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:08.308498 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:08.308459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:08.935269 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:08.935230    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:09.034143 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:09.034102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:09.714507 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:09.714474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:10.385767 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:10.385718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:11.051273 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:11.051236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:11.713786 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:11.713738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:11.762972 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:11.762933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:11.863638 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:11.863607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:12.160349 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.160262    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:12.254986 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.254944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:12.292791 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.292723    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"]
Sep 19 10:52:12.293179 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.293161    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a31e39c7-37aa-451f-bd3b-f68bdbc341e5" containerName="main"
Sep 19 10:52:12.293266 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.293182    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a31e39c7-37aa-451f-bd3b-f68bdbc341e5" containerName="main"
Sep 19 10:52:12.293317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.293303    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a31e39c7-37aa-451f-bd3b-f68bdbc341e5" containerName="main"
Sep 19 10:52:12.293367 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.293317    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ff81ed-6770-4e92-9201-1e36f692392c" containerName="istio-proxy"
Sep 19 10:52:12.297572 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.297546    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"
Sep 19 10:52:12.299809 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.299775    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-e898f8\""
Sep 19 10:52:12.307188 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.307158    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"]
Sep 19 10:52:12.355813 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.355779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd6cb49-7358-442d-93c6-2b19c7b3e02e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s\" (UID: \"2dd6cb49-7358-442d-93c6-2b19c7b3e02e\") " pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"
Sep 19 10:52:12.355813 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.355817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v57cz\" (UniqueName: \"kubernetes.io/projected/2dd6cb49-7358-442d-93c6-2b19c7b3e02e-kube-api-access-v57cz\") pod \"payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s\" (UID: \"2dd6cb49-7358-442d-93c6-2b19c7b3e02e\") " pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"
Sep 19 10:52:12.457121 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.457081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd6cb49-7358-442d-93c6-2b19c7b3e02e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s\" (UID: \"2dd6cb49-7358-442d-93c6-2b19c7b3e02e\") " pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"
Sep 19 10:52:12.457121 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.457125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57cz\" (UniqueName: \"kubernetes.io/projected/2dd6cb49-7358-442d-93c6-2b19c7b3e02e-kube-api-access-v57cz\") pod \"payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s\" (UID: \"2dd6cb49-7358-442d-93c6-2b19c7b3e02e\") " pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"
Sep 19 10:52:12.457946 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.457915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd6cb49-7358-442d-93c6-2b19c7b3e02e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s\" (UID: \"2dd6cb49-7358-442d-93c6-2b19c7b3e02e\") " pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"
Sep 19 10:52:12.464823 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.464787    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57cz\" (UniqueName: \"kubernetes.io/projected/2dd6cb49-7358-442d-93c6-2b19c7b3e02e-kube-api-access-v57cz\") pod \"payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s\" (UID: \"2dd6cb49-7358-442d-93c6-2b19c7b3e02e\") " pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"
Sep 19 10:52:12.608950 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.608914    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"
Sep 19 10:52:12.881966 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:12.881890    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"]
Sep 19 10:52:12.884940 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:52:12.884912    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd6cb49_7358_442d_93c6_2b19c7b3e02e.slice/crio-43550eddfc24a404e3938a701592e955a0938eaff365fcdcc55f77849edfc505 WatchSource:0}: Error finding container 43550eddfc24a404e3938a701592e955a0938eaff365fcdcc55f77849edfc505: Status 404 returned error can't find the container with id 43550eddfc24a404e3938a701592e955a0938eaff365fcdcc55f77849edfc505
Sep 19 10:52:13.445552 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:13.445515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s" event={"ID":"2dd6cb49-7358-442d-93c6-2b19c7b3e02e","Type":"ContainerStarted","Data":"79f1f64de499c7fe594e8ab0a2b6be27c8249935b298451e6441716b14847aa8"}
Sep 19 10:52:13.445998 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:13.445579    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s" event={"ID":"2dd6cb49-7358-442d-93c6-2b19c7b3e02e","Type":"ContainerStarted","Data":"43550eddfc24a404e3938a701592e955a0938eaff365fcdcc55f77849edfc505"}
Sep 19 10:52:13.445998 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:13.445608    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"
Sep 19 10:52:13.462597 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:13.462530    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s" podStartSLOduration=1.462510154 podStartE2EDuration="1.462510154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:52:13.459512722 +0000 UTC m=+978.118257882" watchObservedRunningTime="2026-09-19 10:52:13.462510154 +0000 UTC m=+978.121255315"
Sep 19 10:52:13.870663 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:13.870566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:19.573504 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:19.573412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:19.630884 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:19.630847    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:19.695593 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:19.695545    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:19.803455 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:19.803416    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:20.821712 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:20.821675    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:52:21.747144 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:21.747115    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:24.453162 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:24.453123    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"
Sep 19 10:52:30.263508 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:30.263451    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:31.029462 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:31.029424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:34.545243 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:34.545208    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:35.448233 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:35.448200    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:36.215644 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:36.215612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:36.621064 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:36.620986    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:52:36.921232 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:36.921150    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:37.130080 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:37.130036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:37.234856 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:37.234814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:37.330562 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:37.330520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:37.431086 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:37.431045    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:37.689167 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:37.689117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:38.053954 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:38.053857    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:38.175907 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:38.175866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:38.615021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:38.614986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:39.024012 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:39.023975    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:39.763612 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:39.763568    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:39.913814 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:39.913778    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"]
Sep 19 10:52:39.917477 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:39.917453    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:39.919641 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:39.919619    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1907c1\"/\"premium-w0-1907c1-kserve-self-signed-certs\""
Sep 19 10:52:39.927365 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:39.927339    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"]
Sep 19 10:52:40.000329 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.000292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:40.027715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.027627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwl2m\" (UniqueName: \"kubernetes.io/projected/19ae907f-14b0-49f5-95c2-bededfa382a4-kube-api-access-mwl2m\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.027715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.027671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-model-cache\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.027715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.027701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-tmp-dir\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.027961 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.027813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-dshm\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.027961 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.027845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-home\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.027961 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.027865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19ae907f-14b0-49f5-95c2-bededfa382a4-tls-certs\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.128717 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.128672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-home\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.128717 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.128723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19ae907f-14b0-49f5-95c2-bededfa382a4-tls-certs\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.129025 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.128804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwl2m\" (UniqueName: \"kubernetes.io/projected/19ae907f-14b0-49f5-95c2-bededfa382a4-kube-api-access-mwl2m\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.129025 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.128826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-model-cache\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.129025 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.128848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-tmp-dir\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.129025 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.128899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-dshm\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.129254 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.129154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-model-cache\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.129254 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.129242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-home\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.129337 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.129286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-tmp-dir\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.131487 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.131456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19ae907f-14b0-49f5-95c2-bededfa382a4-tls-certs\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.131589 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.131555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-dshm\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.136126 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.136107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwl2m\" (UniqueName: \"kubernetes.io/projected/19ae907f-14b0-49f5-95c2-bededfa382a4-kube-api-access-mwl2m\") pod \"premium-w0-1907c1-kserve-6c774554f7-g4v76\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") " pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.227845 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.227802    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:40.499778 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.499734    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"]
Sep 19 10:52:40.501959 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:52:40.501933    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19ae907f_14b0_49f5_95c2_bededfa382a4.slice/crio-82c23b4e342d3d96cf08ee47315fdfb8a1c11a0caba9263443acb83a861079fb WatchSource:0}: Error finding container 82c23b4e342d3d96cf08ee47315fdfb8a1c11a0caba9263443acb83a861079fb: Status 404 returned error can't find the container with id 82c23b4e342d3d96cf08ee47315fdfb8a1c11a0caba9263443acb83a861079fb
Sep 19 10:52:40.586019 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.585987    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76" event={"ID":"19ae907f-14b0-49f5-95c2-bededfa382a4","Type":"ContainerStarted","Data":"fb1f590b9b994782abaaf675f7262d4be018c0ca623770cd6ae680da9b7e1edd"}
Sep 19 10:52:40.586019 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.586029    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76" event={"ID":"19ae907f-14b0-49f5-95c2-bededfa382a4","Type":"ContainerStarted","Data":"82c23b4e342d3d96cf08ee47315fdfb8a1c11a0caba9263443acb83a861079fb"}
Sep 19 10:52:40.602274 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:40.602217    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76" podStartSLOduration=1.60219777 podStartE2EDuration="1.60219777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:52:40.600436465 +0000 UTC m=+1005.259181627" watchObservedRunningTime="2026-09-19 10:52:40.60219777 +0000 UTC m=+1005.260942932"
Sep 19 10:52:44.508668 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.508629    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"]
Sep 19 10:52:44.512357 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.512334    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.514428 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.514399    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-df95e5\"/\"openshift-service-ca.crt\""
Sep 19 10:52:44.514547 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.514496    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-df95e5\"/\"test-model-df95e5-kserve-self-signed-certs\""
Sep 19 10:52:44.515206 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.515190    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-df95e5\"/\"kube-root-ca.crt\""
Sep 19 10:52:44.515307 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.515238    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-df95e5\"/\"default-dockercfg-c9mmc\""
Sep 19 10:52:44.522079 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.522054    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"]
Sep 19 10:52:44.532395 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.532364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:44.571319 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.571287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-home\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.571319 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.571329    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-model-cache\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.571551 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.571358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c8428f-76a5-423a-a85a-8a94afd89cf9-tls-certs\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.571551 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.571435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnpz4\" (UniqueName: \"kubernetes.io/projected/a3c8428f-76a5-423a-a85a-8a94afd89cf9-kube-api-access-nnpz4\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.571551 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.571486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-dshm\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.571551 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.571523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-tmp-dir\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.672431 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.672398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-home\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.672431 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.672438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-model-cache\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.672680 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.672464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c8428f-76a5-423a-a85a-8a94afd89cf9-tls-certs\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.672680 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.672508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnpz4\" (UniqueName: \"kubernetes.io/projected/a3c8428f-76a5-423a-a85a-8a94afd89cf9-kube-api-access-nnpz4\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.672680 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.672532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-dshm\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.672680 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.672573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-tmp-dir\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.672937 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.672894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-model-cache\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.672937 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.672924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-home\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.673052 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.673014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-tmp-dir\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.675038 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.675016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-dshm\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.675314 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.675297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c8428f-76a5-423a-a85a-8a94afd89cf9-tls-certs\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.679661 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.679637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnpz4\" (UniqueName: \"kubernetes.io/projected/a3c8428f-76a5-423a-a85a-8a94afd89cf9-kube-api-access-nnpz4\") pod \"test-model-df95e5-kserve-664f7b8f65-kv5l2\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") " pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:44.824208 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:44.824107    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:45.098088 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:45.098013    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"]
Sep 19 10:52:45.100904 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:52:45.100872    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3c8428f_76a5_423a_a85a_8a94afd89cf9.slice/crio-dba82a75f31aed9d183d01685282125b1f1bbe21f12bfef216325dbfa93fdc0f WatchSource:0}: Error finding container dba82a75f31aed9d183d01685282125b1f1bbe21f12bfef216325dbfa93fdc0f: Status 404 returned error can't find the container with id dba82a75f31aed9d183d01685282125b1f1bbe21f12bfef216325dbfa93fdc0f
Sep 19 10:52:45.453132 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:45.453096    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:45.553432 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:45.553397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:45.612355 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:45.612319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2" event={"ID":"a3c8428f-76a5-423a-a85a-8a94afd89cf9","Type":"ContainerStarted","Data":"c1c7fab1c9fd154f8b7ee0492662d8fac04f095580802911403dd3adb4c737e0"}
Sep 19 10:52:45.612355 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:45.612357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2" event={"ID":"a3c8428f-76a5-423a-a85a-8a94afd89cf9","Type":"ContainerStarted","Data":"dba82a75f31aed9d183d01685282125b1f1bbe21f12bfef216325dbfa93fdc0f"}
Sep 19 10:52:45.629742 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:45.629676    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2" podStartSLOduration=1.629658085 podStartE2EDuration="1.629658085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:52:45.627415352 +0000 UTC m=+1010.286160509" watchObservedRunningTime="2026-09-19 10:52:45.629658085 +0000 UTC m=+1010.288403245"
Sep 19 10:52:46.104290 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:46.104247    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:46.162252 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:46.162215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:50.228260 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:50.228216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:50.228654 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:50.228278    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:50.240583 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:50.240561    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:50.646768 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:50.646668    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 10:52:51.876281 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:51.876244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:51.975993 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:51.975958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:52.112898 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:52.112841    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:52.478099 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:52.478065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:52.755398 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:52.755320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:54.824579 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:54.824540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:54.825047 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:54.824600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:54.837351 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:54.837329    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:55.675297 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:55.675267    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:52:57.103880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:57.103845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:57.726231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:57.726194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:57.819560 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:57.819526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:58.617408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:58.617366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:52:59.349157 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:52:59.349117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:00.138056 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:00.138018    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:00.755340 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:00.755306    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:01.318934 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:01.318898    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:53:01.546010 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:01.545979    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:01.667518 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:01.667411    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:01.965784 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:01.965737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:02.182071 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:02.182043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:02.281155 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:02.281068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:17.763892 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:17.763849    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:17.815576 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:17.815533    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:21.310603 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:21.310567    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:53:23.623926 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:23.623875    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:24.557360 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:24.557322    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:25.247543 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:25.247501    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:25.946184 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:25.946156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:26.738279 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:26.738239    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:27.521584 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:27.521549    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:27.814523 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:27.814427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:28.234337 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:28.234302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:28.475017 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:28.474980    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:28.576549 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:28.576457    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:28.891526 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:28.891435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:34.591139 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:34.591098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:34.652198 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:34.652165    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:35.713653 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:35.713612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:36.349400 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:36.349366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:37.011772 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:37.011695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:37.109238 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:37.109201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:37.746050 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:37.746008    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:38.017995 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:38.017915    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:53:38.380997 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:38.380887    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:39.126109 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:39.126073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:39.753674 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:39.753637    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:44.913200 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:44.913167    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"]
Sep 19 10:53:44.916770 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:44.916731    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:44.919804 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:44.919776    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-e898f8\"/\"kube-root-ca.crt\""
Sep 19 10:53:44.919935 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:44.919822    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-e898f8\"/\"openshift-service-ca.crt\""
Sep 19 10:53:44.919935 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:44.919838    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-e898f8\"/\"explicit-model-e898f8-kserve-self-signed-certs\""
Sep 19 10:53:44.919935 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:44.919785    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-e898f8\"/\"default-dockercfg-xffcs\""
Sep 19 10:53:44.927181 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:44.927159    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"]
Sep 19 10:53:44.950920 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:44.950891    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:45.020934 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.020896    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-model-cache\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.021140 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.020965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-tmp-dir\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.021140 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.021033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8cxt\" (UniqueName: \"kubernetes.io/projected/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-kube-api-access-d8cxt\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.021140 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.021065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-dshm\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.021140 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.021089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-home\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.021140 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.021109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-tls-certs\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.121678 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.121640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-tls-certs\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.121918 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.121684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-model-cache\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.121918 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.121745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-tmp-dir\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.121918 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.121864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8cxt\" (UniqueName: \"kubernetes.io/projected/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-kube-api-access-d8cxt\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.122092 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.121983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-dshm\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.122092 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.122034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-home\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.122271 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.122249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-model-cache\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.122333 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.122283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-tmp-dir\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.122382 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.122366    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-home\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.124338 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.124306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-dshm\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.124460 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.124417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-tls-certs\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.129588 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.129564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8cxt\" (UniqueName: \"kubernetes.io/projected/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-kube-api-access-d8cxt\") pod \"explicit-model-e898f8-kserve-dc9467d9-4d2sf\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") " pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.226433 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.226394    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:45.508602 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.508575    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"]
Sep 19 10:53:45.511339 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:53:45.511310    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c1b99be_fe66_4ea1_abc8_52bce5d77b23.slice/crio-e666683cde7abefdb27d7b261d52d599a110b829c5e83739333e08e955314d74 WatchSource:0}: Error finding container e666683cde7abefdb27d7b261d52d599a110b829c5e83739333e08e955314d74: Status 404 returned error can't find the container with id e666683cde7abefdb27d7b261d52d599a110b829c5e83739333e08e955314d74
Sep 19 10:53:45.592376 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.592143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:45.665262 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.665226    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:45.719521 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.719472    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:45.785607 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.785511    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:45.840929 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.840902    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:45.922084 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.922051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf" event={"ID":"6c1b99be-fe66-4ea1-abc8-52bce5d77b23","Type":"ContainerStarted","Data":"cd3144da56f9c251b79a6e527919977bd1951f9a77538aa885555a3278abc465"}
Sep 19 10:53:45.922084 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.922088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf" event={"ID":"6c1b99be-fe66-4ea1-abc8-52bce5d77b23","Type":"ContainerStarted","Data":"e666683cde7abefdb27d7b261d52d599a110b829c5e83739333e08e955314d74"}
Sep 19 10:53:45.938186 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:45.938139    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf" podStartSLOduration=1.938125469 podStartE2EDuration="1.938125469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:53:45.9365361 +0000 UTC m=+1070.595281263" watchObservedRunningTime="2026-09-19 10:53:45.938125469 +0000 UTC m=+1070.596870667"
Sep 19 10:53:46.133020 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:46.132924    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:46.233569 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:46.233539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:47.020198 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:47.020109    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:47.785950 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:47.785915    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:48.409655 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:48.409626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:49.035261 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:49.035228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:49.812207 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:49.812178    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:51.876989 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:51.876938    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:51.977494 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:51.977453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:52.125736 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:52.125696    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:52.447108 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:52.447071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:52.558453 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:52.558407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:52.679330 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:52.679295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:55.226646 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:55.226610    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:55.227215 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:55.226664    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:55.239504 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:55.239469    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:55.984108 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:55.984072    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:53:58.157417 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:58.157381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:58.260630 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:58.260579    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:58.364179 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:58.364137    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:58.499441 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:58.499407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:53:59.432331 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:53:59.432294    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:54:04.191666 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:04.191619    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:04.291804 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:04.291748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:05.078557 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:05.078521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:05.381539 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:05.381443    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:05.568436 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:05.568402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:05.913485 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:05.913455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:07.882938 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:07.882903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:08.526319 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:08.526278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:08.713698 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:08.713657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:08.813790 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:08.813692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:09.410835 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:09.410796    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:10.042893 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:10.042862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:10.828562 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:10.828520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:11.528939 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:11.528903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:12.148537 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:12.148497    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:12.204776 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:12.204716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:12.305093 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:12.305063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:12.595444 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:12.595415    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:12.689730 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:12.689695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:13.512302 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:13.512249    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:14.159123 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:14.159085    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:15.123002 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:15.122963    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:16.834304 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:16.834275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:17.544671 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:17.544635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:18.324055 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:18.324020    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:18.978189 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:18.978156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:19.496846 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:19.496813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:19.695363 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:19.695328    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:20.424497 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:20.424464    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:21.228525 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:21.228487    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:21.941032 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:21.940994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:22.639560 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:22.639528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:23.328642 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:23.328603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:23.435125 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:23.435084    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:23.754479 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:23.754452    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:24.133975 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:24.133886    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:24.240341 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:24.240299    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:24.341081 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:24.341044    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:24.948946 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:24.948911    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:25.804710 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:25.804681    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:26.591093 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:26.591054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:27.146363 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.146314    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:27.200079 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.200046    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"]
Sep 19 10:54:27.200928 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.200294    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s" podUID="2dd6cb49-7358-442d-93c6-2b19c7b3e02e" containerName="payload-processing" containerID="cri-o://79f1f64de499c7fe594e8ab0a2b6be27c8249935b298451e6441716b14847aa8" gracePeriod=30
Sep 19 10:54:27.223652 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.223621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:27.298385 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.298343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:27.379088 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.379053    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:27.436804 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.436744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:27.464932 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.464904    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"
Sep 19 10:54:27.503797 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.503742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:27.534083 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.534054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v57cz\" (UniqueName: \"kubernetes.io/projected/2dd6cb49-7358-442d-93c6-2b19c7b3e02e-kube-api-access-v57cz\") pod \"2dd6cb49-7358-442d-93c6-2b19c7b3e02e\" (UID: \"2dd6cb49-7358-442d-93c6-2b19c7b3e02e\") "
Sep 19 10:54:27.534270 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.534193    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd6cb49-7358-442d-93c6-2b19c7b3e02e-plugins-config-volume\") pod \"2dd6cb49-7358-442d-93c6-2b19c7b3e02e\" (UID: \"2dd6cb49-7358-442d-93c6-2b19c7b3e02e\") "
Sep 19 10:54:27.534552 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.534529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd6cb49-7358-442d-93c6-2b19c7b3e02e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2dd6cb49-7358-442d-93c6-2b19c7b3e02e" (UID: "2dd6cb49-7358-442d-93c6-2b19c7b3e02e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:54:27.536444 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.536424    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd6cb49-7358-442d-93c6-2b19c7b3e02e-kube-api-access-v57cz" (OuterVolumeSpecName: "kube-api-access-v57cz") pod "2dd6cb49-7358-442d-93c6-2b19c7b3e02e" (UID: "2dd6cb49-7358-442d-93c6-2b19c7b3e02e"). InnerVolumeSpecName "kube-api-access-v57cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:54:27.635016 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.634979    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd6cb49-7358-442d-93c6-2b19c7b3e02e-plugins-config-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:54:27.635016 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.635011    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v57cz\" (UniqueName: \"kubernetes.io/projected/2dd6cb49-7358-442d-93c6-2b19c7b3e02e-kube-api-access-v57cz\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:54:27.722202 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:27.722104    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:28.140072 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:28.139983    2580 generic.go:356] "Generic (PLEG): container finished" podID="2dd6cb49-7358-442d-93c6-2b19c7b3e02e" containerID="79f1f64de499c7fe594e8ab0a2b6be27c8249935b298451e6441716b14847aa8" exitCode=0
Sep 19 10:54:28.140072 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:28.140024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s" event={"ID":"2dd6cb49-7358-442d-93c6-2b19c7b3e02e","Type":"ContainerDied","Data":"79f1f64de499c7fe594e8ab0a2b6be27c8249935b298451e6441716b14847aa8"}
Sep 19 10:54:28.140072 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:28.140055    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"
Sep 19 10:54:28.140072 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:28.140070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s" event={"ID":"2dd6cb49-7358-442d-93c6-2b19c7b3e02e","Type":"ContainerDied","Data":"43550eddfc24a404e3938a701592e955a0938eaff365fcdcc55f77849edfc505"}
Sep 19 10:54:28.140403 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:28.140091    2580 scope.go:117] "RemoveContainer" containerID="79f1f64de499c7fe594e8ab0a2b6be27c8249935b298451e6441716b14847aa8"
Sep 19 10:54:28.149043 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:28.149025    2580 scope.go:117] "RemoveContainer" containerID="79f1f64de499c7fe594e8ab0a2b6be27c8249935b298451e6441716b14847aa8"
Sep 19 10:54:28.149307 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:54:28.149290    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f1f64de499c7fe594e8ab0a2b6be27c8249935b298451e6441716b14847aa8\": container with ID starting with 79f1f64de499c7fe594e8ab0a2b6be27c8249935b298451e6441716b14847aa8 not found: ID does not exist" containerID="79f1f64de499c7fe594e8ab0a2b6be27c8249935b298451e6441716b14847aa8"
Sep 19 10:54:28.149354 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:28.149315    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f1f64de499c7fe594e8ab0a2b6be27c8249935b298451e6441716b14847aa8"} err="failed to get container status \"79f1f64de499c7fe594e8ab0a2b6be27c8249935b298451e6441716b14847aa8\": rpc error: code = NotFound desc = could not find container \"79f1f64de499c7fe594e8ab0a2b6be27c8249935b298451e6441716b14847aa8\": container with ID starting with 79f1f64de499c7fe594e8ab0a2b6be27c8249935b298451e6441716b14847aa8 not found: ID does not exist"
Sep 19 10:54:28.156079 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:28.156056    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"]
Sep 19 10:54:28.162182 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:28.162159    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e898f8-c4f657c6f-rbt6s"]
Sep 19 10:54:28.227365 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:28.227326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:28.916221 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:28.916182    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:29.731036 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:29.731003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:29.856689 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:29.856652    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd6cb49-7358-442d-93c6-2b19c7b3e02e" path="/var/lib/kubelet/pods/2dd6cb49-7358-442d-93c6-2b19c7b3e02e/volumes"
Sep 19 10:54:30.541429 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:30.541401    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:31.230738 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:31.230698    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:32.046421 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:32.046384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:32.709050 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:32.709014    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"]
Sep 19 10:54:32.758140 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:32.758077    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:32.816835 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:32.816783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:33.111675 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.111591    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:33.232414 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.232380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:33.328875 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.328845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:33.329527 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.329508    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"]
Sep 19 10:54:33.329919 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.329907    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd6cb49-7358-442d-93c6-2b19c7b3e02e" containerName="payload-processing"
Sep 19 10:54:33.329975 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.329921    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd6cb49-7358-442d-93c6-2b19c7b3e02e" containerName="payload-processing"
Sep 19 10:54:33.330014 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.329981    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd6cb49-7358-442d-93c6-2b19c7b3e02e" containerName="payload-processing"
Sep 19 10:54:33.334253 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.334240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"
Sep 19 10:54:33.336338 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.336318    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-8ddce5\""
Sep 19 10:54:33.336436 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.336395    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-8ddce5-dockercfg-9g5qs\""
Sep 19 10:54:33.340443 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.340422    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"]
Sep 19 10:54:33.373857 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.373786    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"]
Sep 19 10:54:33.377082 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.377065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"
Sep 19 10:54:33.386702 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.386669    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"]
Sep 19 10:54:33.389460 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.389435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k44q4\" (UniqueName: \"kubernetes.io/projected/971fea77-05d6-4ce7-b005-85f93a5f727d-kube-api-access-k44q4\") pod \"payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r\" (UID: \"971fea77-05d6-4ce7-b005-85f93a5f727d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"
Sep 19 10:54:33.389570 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.389496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971fea77-05d6-4ce7-b005-85f93a5f727d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r\" (UID: \"971fea77-05d6-4ce7-b005-85f93a5f727d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"
Sep 19 10:54:33.490795 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.490730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971fea77-05d6-4ce7-b005-85f93a5f727d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r\" (UID: \"971fea77-05d6-4ce7-b005-85f93a5f727d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"
Sep 19 10:54:33.490967 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.490853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195d93e0-1853-490e-8d78-e48fc35a4021-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h\" (UID: \"195d93e0-1853-490e-8d78-e48fc35a4021\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"
Sep 19 10:54:33.490967 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.490875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k44q4\" (UniqueName: \"kubernetes.io/projected/971fea77-05d6-4ce7-b005-85f93a5f727d-kube-api-access-k44q4\") pod \"payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r\" (UID: \"971fea77-05d6-4ce7-b005-85f93a5f727d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"
Sep 19 10:54:33.490967 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.490913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtlp\" (UniqueName: \"kubernetes.io/projected/195d93e0-1853-490e-8d78-e48fc35a4021-kube-api-access-pwtlp\") pod \"payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h\" (UID: \"195d93e0-1853-490e-8d78-e48fc35a4021\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"
Sep 19 10:54:33.491390 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.491372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971fea77-05d6-4ce7-b005-85f93a5f727d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r\" (UID: \"971fea77-05d6-4ce7-b005-85f93a5f727d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"
Sep 19 10:54:33.499420 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.499393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k44q4\" (UniqueName: \"kubernetes.io/projected/971fea77-05d6-4ce7-b005-85f93a5f727d-kube-api-access-k44q4\") pod \"payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r\" (UID: \"971fea77-05d6-4ce7-b005-85f93a5f727d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"
Sep 19 10:54:33.591859 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.591819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtlp\" (UniqueName: \"kubernetes.io/projected/195d93e0-1853-490e-8d78-e48fc35a4021-kube-api-access-pwtlp\") pod \"payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h\" (UID: \"195d93e0-1853-490e-8d78-e48fc35a4021\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"
Sep 19 10:54:33.592056 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.591923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195d93e0-1853-490e-8d78-e48fc35a4021-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h\" (UID: \"195d93e0-1853-490e-8d78-e48fc35a4021\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"
Sep 19 10:54:33.592562 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.592542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195d93e0-1853-490e-8d78-e48fc35a4021-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h\" (UID: \"195d93e0-1853-490e-8d78-e48fc35a4021\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"
Sep 19 10:54:33.599460 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.599435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtlp\" (UniqueName: \"kubernetes.io/projected/195d93e0-1853-490e-8d78-e48fc35a4021-kube-api-access-pwtlp\") pod \"payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h\" (UID: \"195d93e0-1853-490e-8d78-e48fc35a4021\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"
Sep 19 10:54:33.640240 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.640148    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:33.645944 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.645921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"
Sep 19 10:54:33.687981 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.687952    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"
Sep 19 10:54:33.925458 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.925420    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"]
Sep 19 10:54:33.927691 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:54:33.927664    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971fea77_05d6_4ce7_b005_85f93a5f727d.slice/crio-7a7439ba60e79e25a24cab2757ea20e744809d0d1102570c5d61ad101c1084b6 WatchSource:0}: Error finding container 7a7439ba60e79e25a24cab2757ea20e744809d0d1102570c5d61ad101c1084b6: Status 404 returned error can't find the container with id 7a7439ba60e79e25a24cab2757ea20e744809d0d1102570c5d61ad101c1084b6
Sep 19 10:54:33.974315 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:33.974294    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"]
Sep 19 10:54:33.976654 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:54:33.976632    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195d93e0_1853_490e_8d78_e48fc35a4021.slice/crio-dc04d167473612b658f722416dd9678d4cc3e8b21663711e5177dbcace7e77e1 WatchSource:0}: Error finding container dc04d167473612b658f722416dd9678d4cc3e8b21663711e5177dbcace7e77e1: Status 404 returned error can't find the container with id dc04d167473612b658f722416dd9678d4cc3e8b21663711e5177dbcace7e77e1
Sep 19 10:54:34.182392 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:34.182360    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h" event={"ID":"195d93e0-1853-490e-8d78-e48fc35a4021","Type":"ContainerStarted","Data":"4da5f86c318e37c11742f7aecea31795bf83c7ba43bf51aba1e14511ba2fe02d"}
Sep 19 10:54:34.182877 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:34.182414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h" event={"ID":"195d93e0-1853-490e-8d78-e48fc35a4021","Type":"ContainerStarted","Data":"dc04d167473612b658f722416dd9678d4cc3e8b21663711e5177dbcace7e77e1"}
Sep 19 10:54:34.182877 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:34.182479    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"
Sep 19 10:54:34.184296 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:34.184271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r" event={"ID":"971fea77-05d6-4ce7-b005-85f93a5f727d","Type":"ContainerStarted","Data":"ae990377c792c2b4dc17556ea9c7ca84957390255d831713eec94faa26b5dce3"}
Sep 19 10:54:34.184408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:34.184312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r" event={"ID":"971fea77-05d6-4ce7-b005-85f93a5f727d","Type":"ContainerStarted","Data":"7a7439ba60e79e25a24cab2757ea20e744809d0d1102570c5d61ad101c1084b6"}
Sep 19 10:54:34.184408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:34.184332    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"
Sep 19 10:54:34.198146 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:34.198095    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h" podStartSLOduration=1.198080249 podStartE2EDuration="1.198080249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:54:34.196480251 +0000 UTC m=+1118.855225413" watchObservedRunningTime="2026-09-19 10:54:34.198080249 +0000 UTC m=+1118.856825405"
Sep 19 10:54:34.210077 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:34.210023    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r" podStartSLOduration=1.2100055 podStartE2EDuration="1.2100055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:54:34.209491943 +0000 UTC m=+1118.868237104" watchObservedRunningTime="2026-09-19 10:54:34.2100055 +0000 UTC m=+1118.868750663"
Sep 19 10:54:34.711936 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:34.711898    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:35.338523 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:35.338486    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:37.524134 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:37.524095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:54:45.191428 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:45.191396    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"
Sep 19 10:54:45.191836 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:45.191467    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"
Sep 19 10:54:47.729761 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:47.729722    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf" podUID="6c1b99be-fe66-4ea1-abc8-52bce5d77b23" containerName="main" containerID="cri-o://cd3144da56f9c251b79a6e527919977bd1951f9a77538aa885555a3278abc465" gracePeriod=45
Sep 19 10:54:47.987360 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:47.987297    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:54:48.133120 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.133089    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-tmp-dir\") pod \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") "
Sep 19 10:54:48.133299 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.133138    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8cxt\" (UniqueName: \"kubernetes.io/projected/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-kube-api-access-d8cxt\") pod \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") "
Sep 19 10:54:48.133299 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.133195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-model-cache\") pod \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") "
Sep 19 10:54:48.133299 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.133233    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-dshm\") pod \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") "
Sep 19 10:54:48.133299 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.133252    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-tls-certs\") pod \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") "
Sep 19 10:54:48.133299 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.133278    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-home\") pod \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\" (UID: \"6c1b99be-fe66-4ea1-abc8-52bce5d77b23\") "
Sep 19 10:54:48.133562 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.133330    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c1b99be-fe66-4ea1-abc8-52bce5d77b23" (UID: "6c1b99be-fe66-4ea1-abc8-52bce5d77b23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:54:48.133562 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.133499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c1b99be-fe66-4ea1-abc8-52bce5d77b23" (UID: "6c1b99be-fe66-4ea1-abc8-52bce5d77b23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:54:48.133562 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.133530    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-home" (OuterVolumeSpecName: "home") pod "6c1b99be-fe66-4ea1-abc8-52bce5d77b23" (UID: "6c1b99be-fe66-4ea1-abc8-52bce5d77b23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:54:48.133677 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.133653    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:54:48.133677 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.133668    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:54:48.133677 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.133676    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:54:48.135645 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.135616    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-dshm" (OuterVolumeSpecName: "dshm") pod "6c1b99be-fe66-4ea1-abc8-52bce5d77b23" (UID: "6c1b99be-fe66-4ea1-abc8-52bce5d77b23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:54:48.135773 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.135713    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-kube-api-access-d8cxt" (OuterVolumeSpecName: "kube-api-access-d8cxt") pod "6c1b99be-fe66-4ea1-abc8-52bce5d77b23" (UID: "6c1b99be-fe66-4ea1-abc8-52bce5d77b23"). InnerVolumeSpecName "kube-api-access-d8cxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:54:48.135773 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.135720    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c1b99be-fe66-4ea1-abc8-52bce5d77b23" (UID: "6c1b99be-fe66-4ea1-abc8-52bce5d77b23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:54:48.234417 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.234374    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8cxt\" (UniqueName: \"kubernetes.io/projected/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-kube-api-access-d8cxt\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:54:48.234417 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.234409    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:54:48.234417 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.234426    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1b99be-fe66-4ea1-abc8-52bce5d77b23-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:54:48.263584 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.263486    2580 generic.go:356] "Generic (PLEG): container finished" podID="6c1b99be-fe66-4ea1-abc8-52bce5d77b23" containerID="cd3144da56f9c251b79a6e527919977bd1951f9a77538aa885555a3278abc465" exitCode=0
Sep 19 10:54:48.263584 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.263557    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"
Sep 19 10:54:48.263832 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.263575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf" event={"ID":"6c1b99be-fe66-4ea1-abc8-52bce5d77b23","Type":"ContainerDied","Data":"cd3144da56f9c251b79a6e527919977bd1951f9a77538aa885555a3278abc465"}
Sep 19 10:54:48.263832 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.263633    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf" event={"ID":"6c1b99be-fe66-4ea1-abc8-52bce5d77b23","Type":"ContainerDied","Data":"e666683cde7abefdb27d7b261d52d599a110b829c5e83739333e08e955314d74"}
Sep 19 10:54:48.263832 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.263655    2580 scope.go:117] "RemoveContainer" containerID="cd3144da56f9c251b79a6e527919977bd1951f9a77538aa885555a3278abc465"
Sep 19 10:54:48.273500 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.273475    2580 scope.go:117] "RemoveContainer" containerID="cd3144da56f9c251b79a6e527919977bd1951f9a77538aa885555a3278abc465"
Sep 19 10:54:48.273882 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:54:48.273858    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3144da56f9c251b79a6e527919977bd1951f9a77538aa885555a3278abc465\": container with ID starting with cd3144da56f9c251b79a6e527919977bd1951f9a77538aa885555a3278abc465 not found: ID does not exist" containerID="cd3144da56f9c251b79a6e527919977bd1951f9a77538aa885555a3278abc465"
Sep 19 10:54:48.274014 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.273895    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3144da56f9c251b79a6e527919977bd1951f9a77538aa885555a3278abc465"} err="failed to get container status \"cd3144da56f9c251b79a6e527919977bd1951f9a77538aa885555a3278abc465\": rpc error: code = NotFound desc = could not find container \"cd3144da56f9c251b79a6e527919977bd1951f9a77538aa885555a3278abc465\": container with ID starting with cd3144da56f9c251b79a6e527919977bd1951f9a77538aa885555a3278abc465 not found: ID does not exist"
Sep 19 10:54:48.284803 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.284775    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"]
Sep 19 10:54:48.286603 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:48.286581    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-e898f8/explicit-model-e898f8-kserve-dc9467d9-4d2sf"]
Sep 19 10:54:49.857581 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:49.857548    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c1b99be-fe66-4ea1-abc8-52bce5d77b23" path="/var/lib/kubelet/pods/6c1b99be-fe66-4ea1-abc8-52bce5d77b23/volumes"
Sep 19 10:54:53.624311 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:53.624271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:55.121950 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.121918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:55.802460 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.802432    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"]
Sep 19 10:54:55.802901 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.802884    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1b99be-fe66-4ea1-abc8-52bce5d77b23" containerName="main"
Sep 19 10:54:55.803006 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.802903    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1b99be-fe66-4ea1-abc8-52bce5d77b23" containerName="main"
Sep 19 10:54:55.803006 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.802998    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1b99be-fe66-4ea1-abc8-52bce5d77b23" containerName="main"
Sep 19 10:54:55.806116 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.806096    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:55.808113 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.808095    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5116ec\"/\"e2e-conflict-a-9f7696ca-kserve-self-signed-certs\""
Sep 19 10:54:55.814285 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.814265    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"]
Sep 19 10:54:55.906487 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.906451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhzxn\" (UniqueName: \"kubernetes.io/projected/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-kube-api-access-fhzxn\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:55.906664 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.906498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-model-cache\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:55.906664 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.906580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-dshm\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:55.906664 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.906647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-tmp-dir\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:55.906826 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.906682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-home\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:55.906826 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.906696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-tls-certs\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:55.940090 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:55.940061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:56.007279 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.007248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzxn\" (UniqueName: \"kubernetes.io/projected/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-kube-api-access-fhzxn\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:56.007454 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.007289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-model-cache\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:56.007454 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.007337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-dshm\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:56.007454 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.007375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-tmp-dir\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:56.007454 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.007418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-home\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:56.007454 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.007441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-tls-certs\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:56.007816 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.007792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-home\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:56.007890 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.007796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-model-cache\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:56.007890 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.007836    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-tmp-dir\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:56.009855 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.009832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-dshm\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:56.010036 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.010018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-tls-certs\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:56.015646 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.015624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhzxn\" (UniqueName: \"kubernetes.io/projected/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-kube-api-access-fhzxn\") pod \"e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") " pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:56.041704 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.041671    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:56.115562 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.115454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:54:56.392718 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.392693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"]
Sep 19 10:54:56.394578 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:54:56.394546    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa3ff5bf_d397_4760_9ee3_e235b143e6a5.slice/crio-35dfd927302ab6b53afd1a036425cf6e6cfafe290202156432d3c84702991816 WatchSource:0}: Error finding container 35dfd927302ab6b53afd1a036425cf6e6cfafe290202156432d3c84702991816: Status 404 returned error can't find the container with id 35dfd927302ab6b53afd1a036425cf6e6cfafe290202156432d3c84702991816
Sep 19 10:54:56.811744 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:56.811707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:57.110136 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:57.110051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:57.314968 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:57.314932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn" event={"ID":"fa3ff5bf-d397-4760-9ee3-e235b143e6a5","Type":"ContainerStarted","Data":"111e7f526963e6a7b712e39567b5b85cb0b7f36f9c39d10493b4c51d37e48d72"}
Sep 19 10:54:57.315136 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:57.314977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn" event={"ID":"fa3ff5bf-d397-4760-9ee3-e235b143e6a5","Type":"ContainerStarted","Data":"35dfd927302ab6b53afd1a036425cf6e6cfafe290202156432d3c84702991816"}
Sep 19 10:54:57.639566 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:57.639518    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:58.393214 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:58.393172    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:59.089408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:59.089363    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:54:59.725155 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:54:59.725119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:00.019861 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:00.019748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:06.115645 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:06.115610    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:55:06.116173 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:06.115667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:55:06.130129 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:06.130104    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:55:06.147486 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:06.147426    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn" podStartSLOduration=11.147405147 podStartE2EDuration="11.147405147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:54:57.331057458 +0000 UTC m=+1141.989802618" watchObservedRunningTime="2026-09-19 10:55:06.147405147 +0000 UTC m=+1150.806150330"
Sep 19 10:55:06.377412 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:06.377315    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:55:08.356498 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:08.356454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:08.649226 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:08.649142    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:08.913502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:08.913409    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:08.928860 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:08.928826    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:55:09.939685 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:09.939646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:10.175386 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:10.175352    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:10.607175 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:10.607139    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:10.709569 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:10.709523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:10.904543 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:10.904454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:11.251608 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:11.251567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:11.389189 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:11.389138    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:11.455815 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:11.455777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:11.725156 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:11.725121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:12.539160 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:12.539120    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:12.634257 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:12.634221    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:12.735577 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:12.735503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:13.336441 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:13.336403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:13.975076 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:13.975038    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:14.721408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:14.721375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:15.450341 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:15.450303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:16.082531 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.082492    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"]
Sep 19 10:55:16.087289 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.087271    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.090507 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.090486    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9ebfd604-serving-cert\""
Sep 19 10:55:16.105989 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.105959    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"]
Sep 19 10:55:16.138894 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.138863    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:16.204611 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.204571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-689tw\" (UniqueName: \"kubernetes.io/projected/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-kube-api-access-689tw\") pod \"maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.204825 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.204698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-maas-api-tls\") pod \"maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.204825 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.204783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-openshift-service-ca\") pod \"maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.204909 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.204842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-metrics-certs\") pod \"maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.221266 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.221227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:16.305712 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.305677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-689tw\" (UniqueName: \"kubernetes.io/projected/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-kube-api-access-689tw\") pod \"maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.305918 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.305806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-maas-api-tls\") pod \"maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.305918 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.305862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-openshift-service-ca\") pod \"maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.305918 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.305893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-metrics-certs\") pod \"maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.306635 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.306604    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-openshift-service-ca\") pod \"maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.308480 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.308459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-maas-api-tls\") pod \"maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.308602 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.308551    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-metrics-certs\") pod \"maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.314175 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.314145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-689tw\" (UniqueName: \"kubernetes.io/projected/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-kube-api-access-689tw\") pod \"maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.398155 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.398057    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:16.671343 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.671315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"]
Sep 19 10:55:16.673368 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:55:16.673341    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0f0d0f_c34c_4e01_b2c7_4e43700e2925.slice/crio-2488281668fbcdbe855f57321df9609f28356aaee6d64318202d125eb608cf70 WatchSource:0}: Error finding container 2488281668fbcdbe855f57321df9609f28356aaee6d64318202d125eb608cf70: Status 404 returned error can't find the container with id 2488281668fbcdbe855f57321df9609f28356aaee6d64318202d125eb608cf70
Sep 19 10:55:16.674818 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.674799    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 10:55:16.733288 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.733255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:16.823013 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.822980    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:16.831553 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.831521    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"]
Sep 19 10:55:16.836294 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.836275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"
Sep 19 10:55:16.838703 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.838680    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9ebfd604-dockercfg-vwwqr\""
Sep 19 10:55:16.838703 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.838694    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9ebfd604\""
Sep 19 10:55:16.846192 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.846161    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"]
Sep 19 10:55:16.912964 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.912925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6bdf\" (UniqueName: \"kubernetes.io/projected/9573261a-046e-4fd9-9953-37a9f2efeec6-kube-api-access-b6bdf\") pod \"payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr\" (UID: \"9573261a-046e-4fd9-9953-37a9f2efeec6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"
Sep 19 10:55:16.912964 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:16.912965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9573261a-046e-4fd9-9953-37a9f2efeec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr\" (UID: \"9573261a-046e-4fd9-9953-37a9f2efeec6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"
Sep 19 10:55:17.013581 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:17.013533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bdf\" (UniqueName: \"kubernetes.io/projected/9573261a-046e-4fd9-9953-37a9f2efeec6-kube-api-access-b6bdf\") pod \"payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr\" (UID: \"9573261a-046e-4fd9-9953-37a9f2efeec6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"
Sep 19 10:55:17.013581 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:17.013580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9573261a-046e-4fd9-9953-37a9f2efeec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr\" (UID: \"9573261a-046e-4fd9-9953-37a9f2efeec6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"
Sep 19 10:55:17.014270 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:17.014250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9573261a-046e-4fd9-9953-37a9f2efeec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr\" (UID: \"9573261a-046e-4fd9-9953-37a9f2efeec6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"
Sep 19 10:55:17.021108 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:17.021088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bdf\" (UniqueName: \"kubernetes.io/projected/9573261a-046e-4fd9-9953-37a9f2efeec6-kube-api-access-b6bdf\") pod \"payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr\" (UID: \"9573261a-046e-4fd9-9953-37a9f2efeec6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"
Sep 19 10:55:17.149256 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:17.149227    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"
Sep 19 10:55:17.425482 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:17.425375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms" event={"ID":"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925","Type":"ContainerStarted","Data":"d4f9bd9d6a02fcdcf7958cb8618a7b3c7856669c7289532a363c3ba626ebae46"}
Sep 19 10:55:17.425482 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:17.425446    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms" event={"ID":"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925","Type":"ContainerStarted","Data":"2488281668fbcdbe855f57321df9609f28356aaee6d64318202d125eb608cf70"}
Sep 19 10:55:17.425817 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:17.425799    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:17.427542 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:17.427520    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"]
Sep 19 10:55:17.430104 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:55:17.430078    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9573261a_046e_4fd9_9953_37a9f2efeec6.slice/crio-d97f4bc1358f191fae771f54b76a1745aa34507b132f0fbb9d2021c46551c67b WatchSource:0}: Error finding container d97f4bc1358f191fae771f54b76a1745aa34507b132f0fbb9d2021c46551c67b: Status 404 returned error can't find the container with id d97f4bc1358f191fae771f54b76a1745aa34507b132f0fbb9d2021c46551c67b
Sep 19 10:55:17.441229 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:17.441183    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms" podStartSLOduration=0.985231434 podStartE2EDuration="1.441165898s" totalImagesPullingTime="455.934464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:55:17.439557298 +0000 UTC m=+1162.098302459" watchObservedRunningTime="2026-09-19 10:55:17.441165898 +0000 UTC m=+1162.099911058"
Sep 19 10:55:18.432825 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:18.432779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr" event={"ID":"9573261a-046e-4fd9-9953-37a9f2efeec6","Type":"ContainerStarted","Data":"f687250570362e9e681f8767cf8ec96ffa30dfca50f8cc5e57984eaadf9bdb0d"}
Sep 19 10:55:18.433254 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:18.432847    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr" event={"ID":"9573261a-046e-4fd9-9953-37a9f2efeec6","Type":"ContainerStarted","Data":"d97f4bc1358f191fae771f54b76a1745aa34507b132f0fbb9d2021c46551c67b"}
Sep 19 10:55:18.433254 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:18.433098    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"
Sep 19 10:55:18.449178 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:18.449127    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr" podStartSLOduration=2.449110409 podStartE2EDuration="2.449110409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:55:18.446730591 +0000 UTC m=+1163.105475750" watchObservedRunningTime="2026-09-19 10:55:18.449110409 +0000 UTC m=+1163.107855636"
Sep 19 10:55:23.439234 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:23.439202    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:55:29.441237 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:29.441204    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"
Sep 19 10:55:31.717632 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:31.717590    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:55:38.899021 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:38.898975    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:39.616631 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:39.616585    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:40.226622 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:40.226582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:40.838135 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:40.838097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:41.448972 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:41.448936    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:41.587575 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:41.587541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:42.226081 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:42.226038    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:42.932646 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:42.932603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:43.538231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:43.538185    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:55:55.311737 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:55.311699    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:55:55.866481 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:55.866446    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 10:55:55.867350 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:55:55.867326    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 10:56:02.820182 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:02.820149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:13.726228 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:13.726190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:56:15.717986 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:15.717956    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:18.695681 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:18.695639    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"]
Sep 19 10:56:18.714921 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:18.714878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:26.713862 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:26.713816    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:30.628915 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:30.628855    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:31.314152 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:31.314113    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:32.115565 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:32.115511    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:32.738929 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:32.738896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:33.307893 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.307860    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"]
Sep 19 10:56:33.311725 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.311699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.314510 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.314482    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9ebfd604\"/\"default-dockercfg-w96kw\""
Sep 19 10:56:33.314655 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.314491    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9ebfd604\"/\"openshift-service-ca.crt\""
Sep 19 10:56:33.314655 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.314501    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9ebfd604\"/\"e2e-lifecycle-model-9ebfd604-kserve-self-signed-certs\""
Sep 19 10:56:33.314655 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.314511    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9ebfd604\"/\"kube-root-ca.crt\""
Sep 19 10:56:33.321482 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.321457    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"]
Sep 19 10:56:33.337035 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.336990    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:33.384186 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.384149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-home\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.384387 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.384192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxszg\" (UniqueName: \"kubernetes.io/projected/6bee274b-77b5-40c8-b213-bcf52c2eadbb-kube-api-access-gxszg\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.384387 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.384222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-model-cache\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.384387 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.384295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-tmp-dir\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.384387 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.384377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-dshm\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.384552 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.384405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bee274b-77b5-40c8-b213-bcf52c2eadbb-tls-certs\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.485125 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.485090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-dshm\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.485125 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.485128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bee274b-77b5-40c8-b213-bcf52c2eadbb-tls-certs\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.485401 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.485162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-home\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.485401 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.485181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxszg\" (UniqueName: \"kubernetes.io/projected/6bee274b-77b5-40c8-b213-bcf52c2eadbb-kube-api-access-gxszg\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.485401 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.485202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-model-cache\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.485401 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.485231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-tmp-dir\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.485654 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.485626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-home\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.485708 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.485651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-model-cache\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.485745 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.485704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-tmp-dir\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.487724 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.487701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-dshm\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.488011 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.487989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bee274b-77b5-40c8-b213-bcf52c2eadbb-tls-certs\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.492629 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.492603    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxszg\" (UniqueName: \"kubernetes.io/projected/6bee274b-77b5-40c8-b213-bcf52c2eadbb-kube-api-access-gxszg\") pod \"e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") " pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.605095 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.604975    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:33.622317 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.622287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:33.717526 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.717482    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs" podUID="dd962b8a-8a90-42ea-8d4b-9295601dafd8" containerName="main" containerID="cri-o://eeeac578968125a1d02591300ecea7a95dd36db6ffa77d17b48f23a7d8cda301" gracePeriod=45
Sep 19 10:56:33.851061 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.851029    2580 generic.go:356] "Generic (PLEG): container finished" podID="dd962b8a-8a90-42ea-8d4b-9295601dafd8" containerID="eeeac578968125a1d02591300ecea7a95dd36db6ffa77d17b48f23a7d8cda301" exitCode=0
Sep 19 10:56:33.856685 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.856604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs" event={"ID":"dd962b8a-8a90-42ea-8d4b-9295601dafd8","Type":"ContainerDied","Data":"eeeac578968125a1d02591300ecea7a95dd36db6ffa77d17b48f23a7d8cda301"}
Sep 19 10:56:33.905560 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.905528    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"]
Sep 19 10:56:33.907277 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:56:33.907250    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bee274b_77b5_40c8_b213_bcf52c2eadbb.slice/crio-e87b79230aa9feae79a6f82a40cd21e5993042281dd6dcc92f9f79679fc5da91 WatchSource:0}: Error finding container e87b79230aa9feae79a6f82a40cd21e5993042281dd6dcc92f9f79679fc5da91: Status 404 returned error can't find the container with id e87b79230aa9feae79a6f82a40cd21e5993042281dd6dcc92f9f79679fc5da91
Sep 19 10:56:33.965910 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:33.965885    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:56:34.091536 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.091493    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd962b8a-8a90-42ea-8d4b-9295601dafd8-tls-certs\") pod \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") "
Sep 19 10:56:34.091717 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.091551    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-model-cache\") pod \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") "
Sep 19 10:56:34.091717 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.091615    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-tmp-dir\") pod \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") "
Sep 19 10:56:34.091717 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.091684    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbx86\" (UniqueName: \"kubernetes.io/projected/dd962b8a-8a90-42ea-8d4b-9295601dafd8-kube-api-access-nbx86\") pod \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") "
Sep 19 10:56:34.091717 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.091702    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-dshm\") pod \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") "
Sep 19 10:56:34.091947 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.091726    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-home\") pod \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\" (UID: \"dd962b8a-8a90-42ea-8d4b-9295601dafd8\") "
Sep 19 10:56:34.091947 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.091930    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd962b8a-8a90-42ea-8d4b-9295601dafd8" (UID: "dd962b8a-8a90-42ea-8d4b-9295601dafd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:56:34.092056 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.091957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd962b8a-8a90-42ea-8d4b-9295601dafd8" (UID: "dd962b8a-8a90-42ea-8d4b-9295601dafd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:56:34.092112 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.092067    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-home" (OuterVolumeSpecName: "home") pod "dd962b8a-8a90-42ea-8d4b-9295601dafd8" (UID: "dd962b8a-8a90-42ea-8d4b-9295601dafd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:56:34.092112 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.092089    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:34.092112 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.092104    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:34.094079 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.094050    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd962b8a-8a90-42ea-8d4b-9295601dafd8-kube-api-access-nbx86" (OuterVolumeSpecName: "kube-api-access-nbx86") pod "dd962b8a-8a90-42ea-8d4b-9295601dafd8" (UID: "dd962b8a-8a90-42ea-8d4b-9295601dafd8"). InnerVolumeSpecName "kube-api-access-nbx86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:56:34.094079 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.094054    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-dshm" (OuterVolumeSpecName: "dshm") pod "dd962b8a-8a90-42ea-8d4b-9295601dafd8" (UID: "dd962b8a-8a90-42ea-8d4b-9295601dafd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:56:34.094079 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.094055    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd962b8a-8a90-42ea-8d4b-9295601dafd8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd962b8a-8a90-42ea-8d4b-9295601dafd8" (UID: "dd962b8a-8a90-42ea-8d4b-9295601dafd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:56:34.192898 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.192862    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbx86\" (UniqueName: \"kubernetes.io/projected/dd962b8a-8a90-42ea-8d4b-9295601dafd8-kube-api-access-nbx86\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:34.192898 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.192891    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:34.192898 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.192900    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd962b8a-8a90-42ea-8d4b-9295601dafd8-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:34.193150 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.192908    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd962b8a-8a90-42ea-8d4b-9295601dafd8-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:34.217286 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.217228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:34.857421 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.857323    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"
Sep 19 10:56:34.857421 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.857348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs" event={"ID":"dd962b8a-8a90-42ea-8d4b-9295601dafd8","Type":"ContainerDied","Data":"8067a2cce424edf1fad2c6ed3def978ae0251be1b78eccbbd9487372555b81ba"}
Sep 19 10:56:34.857421 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.857402    2580 scope.go:117] "RemoveContainer" containerID="eeeac578968125a1d02591300ecea7a95dd36db6ffa77d17b48f23a7d8cda301"
Sep 19 10:56:34.859507 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.859480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn" event={"ID":"6bee274b-77b5-40c8-b213-bcf52c2eadbb","Type":"ContainerStarted","Data":"7951a74fea23b38d87d1ee6d5a45c62f08fa95469d723e8557d9b1dd4391d4b0"}
Sep 19 10:56:34.859691 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.859676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn" event={"ID":"6bee274b-77b5-40c8-b213-bcf52c2eadbb","Type":"ContainerStarted","Data":"e87b79230aa9feae79a6f82a40cd21e5993042281dd6dcc92f9f79679fc5da91"}
Sep 19 10:56:34.879009 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.878947    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn" podStartSLOduration=1.878924649 podStartE2EDuration="1.878924649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:56:34.876534658 +0000 UTC m=+1239.535279819" watchObservedRunningTime="2026-09-19 10:56:34.878924649 +0000 UTC m=+1239.537669810"
Sep 19 10:56:34.890586 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.890553    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"]
Sep 19 10:56:34.894290 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.894262    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6a0d36/test-model-6a0d36-kserve-77c7488969-5srcs"]
Sep 19 10:56:34.939071 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:34.939029    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:35.719002 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:35.718967    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:35.857186 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:35.857153    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd962b8a-8a90-42ea-8d4b-9295601dafd8" path="/var/lib/kubelet/pods/dd962b8a-8a90-42ea-8d4b-9295601dafd8/volumes"
Sep 19 10:56:36.340816 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:36.340781    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:36.822628 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:36.822586    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"]
Sep 19 10:56:36.839497 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:36.839470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:40.912406 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:40.912366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:40.980198 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:40.980146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:41.077390 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:41.077344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:41.154822 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:41.154789    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:41.448832 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:41.448798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:41.601059 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:41.601022    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:42.236857 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:42.236820    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:42.905510 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:42.905479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:43.614516 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:43.614463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:43.622485 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:43.622447    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:43.622661 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:43.622506    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:43.635783 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:43.635723    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:43.923394 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:43.923304    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:56:44.237569 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:44.237530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:44.843628 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:44.843601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:45.822618 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:45.822579    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:56:45.892257 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:45.892225    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:47.514566 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:47.514487    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:47.618004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:47.617969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:47.755065 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:47.755024    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:48.402525 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:48.402494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:49.036235 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:49.036197    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:49.834354 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:49.834310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:50.515364 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:50.515326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:51.206879 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:51.206845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:51.303474 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:51.303437    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:51.417095 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:51.417055    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:51.529025 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:51.528927    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:51.843446 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:51.843344    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2" podUID="a3c8428f-76a5-423a-a85a-8a94afd89cf9" containerName="main" containerID="cri-o://c1c7fab1c9fd154f8b7ee0492662d8fac04f095580802911403dd3adb4c737e0" gracePeriod=45
Sep 19 10:56:51.976434 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:51.976398    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3c8428f-76a5-423a-a85a-8a94afd89cf9" containerID="c1c7fab1c9fd154f8b7ee0492662d8fac04f095580802911403dd3adb4c737e0" exitCode=0
Sep 19 10:56:51.976635 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:51.976458    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2" event={"ID":"a3c8428f-76a5-423a-a85a-8a94afd89cf9","Type":"ContainerDied","Data":"c1c7fab1c9fd154f8b7ee0492662d8fac04f095580802911403dd3adb4c737e0"}
Sep 19 10:56:52.035026 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.034987    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:52.107950 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.107883    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:56:52.268694 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.268651    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c8428f-76a5-423a-a85a-8a94afd89cf9-tls-certs\") pod \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") "
Sep 19 10:56:52.268914 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.268710    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-tmp-dir\") pod \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") "
Sep 19 10:56:52.268914 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.268784    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-dshm\") pod \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") "
Sep 19 10:56:52.268914 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.268823    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-model-cache\") pod \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") "
Sep 19 10:56:52.269071 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.268927    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-home\") pod \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") "
Sep 19 10:56:52.269071 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.268967    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnpz4\" (UniqueName: \"kubernetes.io/projected/a3c8428f-76a5-423a-a85a-8a94afd89cf9-kube-api-access-nnpz4\") pod \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\" (UID: \"a3c8428f-76a5-423a-a85a-8a94afd89cf9\") "
Sep 19 10:56:52.269071 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.269051    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3c8428f-76a5-423a-a85a-8a94afd89cf9" (UID: "a3c8428f-76a5-423a-a85a-8a94afd89cf9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:56:52.269231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.269161    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3c8428f-76a5-423a-a85a-8a94afd89cf9" (UID: "a3c8428f-76a5-423a-a85a-8a94afd89cf9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:56:52.269231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.269186    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-home" (OuterVolumeSpecName: "home") pod "a3c8428f-76a5-423a-a85a-8a94afd89cf9" (UID: "a3c8428f-76a5-423a-a85a-8a94afd89cf9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:56:52.269404 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.269389    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:52.269451 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.269408    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:52.269451 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.269418    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:52.271686 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.271657    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-dshm" (OuterVolumeSpecName: "dshm") pod "a3c8428f-76a5-423a-a85a-8a94afd89cf9" (UID: "a3c8428f-76a5-423a-a85a-8a94afd89cf9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:56:52.271825 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.271800    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c8428f-76a5-423a-a85a-8a94afd89cf9-kube-api-access-nnpz4" (OuterVolumeSpecName: "kube-api-access-nnpz4") pod "a3c8428f-76a5-423a-a85a-8a94afd89cf9" (UID: "a3c8428f-76a5-423a-a85a-8a94afd89cf9"). InnerVolumeSpecName "kube-api-access-nnpz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:56:52.271825 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.271811    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3c8428f-76a5-423a-a85a-8a94afd89cf9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3c8428f-76a5-423a-a85a-8a94afd89cf9" (UID: "a3c8428f-76a5-423a-a85a-8a94afd89cf9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:56:52.370696 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.370585    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c8428f-76a5-423a-a85a-8a94afd89cf9-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:52.370696 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.370625    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnpz4\" (UniqueName: \"kubernetes.io/projected/a3c8428f-76a5-423a-a85a-8a94afd89cf9-kube-api-access-nnpz4\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:52.370696 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.370643    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c8428f-76a5-423a-a85a-8a94afd89cf9-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:52.712029 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.711958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:52.983127 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.983033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2" event={"ID":"a3c8428f-76a5-423a-a85a-8a94afd89cf9","Type":"ContainerDied","Data":"dba82a75f31aed9d183d01685282125b1f1bbe21f12bfef216325dbfa93fdc0f"}
Sep 19 10:56:52.983127 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.983073    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"
Sep 19 10:56:52.983127 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:52.983089    2580 scope.go:117] "RemoveContainer" containerID="c1c7fab1c9fd154f8b7ee0492662d8fac04f095580802911403dd3adb4c737e0"
Sep 19 10:56:53.005916 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:53.005882    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"]
Sep 19 10:56:53.008608 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:53.008578    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-df95e5/test-model-df95e5-kserve-664f7b8f65-kv5l2"]
Sep 19 10:56:53.335028 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:53.334912    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:53.856775 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:53.856725    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3c8428f-76a5-423a-a85a-8a94afd89cf9" path="/var/lib/kubelet/pods/a3c8428f-76a5-423a-a85a-8a94afd89cf9/volumes"
Sep 19 10:56:55.339302 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:55.339263    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:56.559136 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:56.559093    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:58.109365 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.109330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:58.480859 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.480821    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"]
Sep 19 10:56:58.481131 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.481100    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m" podUID="2017702a-00f7-487f-babd-0057c8fdcbed" containerName="maas-api" containerID="cri-o://25793f1db31541227570da12454f357f0d50e749642167ae7f39ee876fc137f0" gracePeriod=30
Sep 19 10:56:58.518914 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.518880    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:58.579797 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.579740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:58.678502 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.678466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:58.736398 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.736332    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:56:58.758809 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.758720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:58.831214 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.831181    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-maas-api-tls\") pod \"2017702a-00f7-487f-babd-0057c8fdcbed\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") "
Sep 19 10:56:58.831403 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.831270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2017702a-00f7-487f-babd-0057c8fdcbed-openshift-service-ca\") pod \"2017702a-00f7-487f-babd-0057c8fdcbed\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") "
Sep 19 10:56:58.831403 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.831289    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-metrics-certs\") pod \"2017702a-00f7-487f-babd-0057c8fdcbed\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") "
Sep 19 10:56:58.831550 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.831455    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d74ck\" (UniqueName: \"kubernetes.io/projected/2017702a-00f7-487f-babd-0057c8fdcbed-kube-api-access-d74ck\") pod \"2017702a-00f7-487f-babd-0057c8fdcbed\" (UID: \"2017702a-00f7-487f-babd-0057c8fdcbed\") "
Sep 19 10:56:58.831633 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.831608    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2017702a-00f7-487f-babd-0057c8fdcbed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2017702a-00f7-487f-babd-0057c8fdcbed" (UID: "2017702a-00f7-487f-babd-0057c8fdcbed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:56:58.831824 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.831802    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2017702a-00f7-487f-babd-0057c8fdcbed-openshift-service-ca\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:58.833611 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.833587    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2017702a-00f7-487f-babd-0057c8fdcbed" (UID: "2017702a-00f7-487f-babd-0057c8fdcbed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:56:58.833611 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.833601    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2017702a-00f7-487f-babd-0057c8fdcbed-kube-api-access-d74ck" (OuterVolumeSpecName: "kube-api-access-d74ck") pod "2017702a-00f7-487f-babd-0057c8fdcbed" (UID: "2017702a-00f7-487f-babd-0057c8fdcbed"). InnerVolumeSpecName "kube-api-access-d74ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:56:58.833788 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.833722    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2017702a-00f7-487f-babd-0057c8fdcbed" (UID: "2017702a-00f7-487f-babd-0057c8fdcbed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:56:58.933268 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.933214    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-metrics-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:58.933268 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.933251    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d74ck\" (UniqueName: \"kubernetes.io/projected/2017702a-00f7-487f-babd-0057c8fdcbed-kube-api-access-d74ck\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:58.933268 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:58.933263    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2017702a-00f7-487f-babd-0057c8fdcbed-maas-api-tls\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:56:59.016471 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:59.016374    2580 generic.go:356] "Generic (PLEG): container finished" podID="2017702a-00f7-487f-babd-0057c8fdcbed" containerID="25793f1db31541227570da12454f357f0d50e749642167ae7f39ee876fc137f0" exitCode=0
Sep 19 10:56:59.016471 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:59.016426    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"
Sep 19 10:56:59.016714 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:59.016468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m" event={"ID":"2017702a-00f7-487f-babd-0057c8fdcbed","Type":"ContainerDied","Data":"25793f1db31541227570da12454f357f0d50e749642167ae7f39ee876fc137f0"}
Sep 19 10:56:59.016714 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:59.016538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m" event={"ID":"2017702a-00f7-487f-babd-0057c8fdcbed","Type":"ContainerDied","Data":"7529ff82366a591c1865d51418325c30f387109eb57ad50b0523f9780a2a3f2b"}
Sep 19 10:56:59.016714 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:59.016572    2580 scope.go:117] "RemoveContainer" containerID="25793f1db31541227570da12454f357f0d50e749642167ae7f39ee876fc137f0"
Sep 19 10:56:59.025794 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:59.025775    2580 scope.go:117] "RemoveContainer" containerID="25793f1db31541227570da12454f357f0d50e749642167ae7f39ee876fc137f0"
Sep 19 10:56:59.026110 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:56:59.026088    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25793f1db31541227570da12454f357f0d50e749642167ae7f39ee876fc137f0\": container with ID starting with 25793f1db31541227570da12454f357f0d50e749642167ae7f39ee876fc137f0 not found: ID does not exist" containerID="25793f1db31541227570da12454f357f0d50e749642167ae7f39ee876fc137f0"
Sep 19 10:56:59.026181 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:59.026120    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25793f1db31541227570da12454f357f0d50e749642167ae7f39ee876fc137f0"} err="failed to get container status \"25793f1db31541227570da12454f357f0d50e749642167ae7f39ee876fc137f0\": rpc error: code = NotFound desc = could not find container \"25793f1db31541227570da12454f357f0d50e749642167ae7f39ee876fc137f0\": container with ID starting with 25793f1db31541227570da12454f357f0d50e749642167ae7f39ee876fc137f0 not found: ID does not exist"
Sep 19 10:56:59.037218 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:59.037186    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"]
Sep 19 10:56:59.040325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:59.040298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:59.040664 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:59.040646    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-df95e5-6ddbb67b87-bps8m"]
Sep 19 10:56:59.172314 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:59.172283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:59.807740 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:59.807703    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:56:59.861298 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:56:59.861259    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2017702a-00f7-487f-babd-0057c8fdcbed" path="/var/lib/kubelet/pods/2017702a-00f7-487f-babd-0057c8fdcbed/volumes"
Sep 19 10:57:00.618140 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:00.618102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:01.250420 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:01.250382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:02.007254 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:02.007213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:02.624535 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:02.624493    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:04.629785 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:04.629723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:04.693605 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:04.693573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:04.855432 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:04.855394    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"]
Sep 19 10:57:04.855687 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:04.855660    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerName="istio-proxy" containerID="cri-o://f4a02b0be6984ccc1bb58c5b268bfde35355fca4c40603dfac6f57f3cbb7442b" gracePeriod=30
Sep 19 10:57:04.869706 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:04.869672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:05.418544 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:05.418504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:05.710943 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:05.710908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:06.190605 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:06.190573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:06.803799 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:06.803742    2580 patch_prober.go:28] interesting pod/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 19 10:57:06.804184 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:06.803837    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 10:57:08.040013 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:08.039973    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:08.667398 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:08.667352    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:08.785571 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:08.785529    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:08.883978 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:08.883933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:09.429329 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:09.429275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:10.078490 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.078454    2580 generic.go:356] "Generic (PLEG): container finished" podID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerID="f4a02b0be6984ccc1bb58c5b268bfde35355fca4c40603dfac6f57f3cbb7442b" exitCode=0
Sep 19 10:57:10.078695 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.078522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" event={"ID":"caa42699-c4cd-41a8-8825-61cb1b057d42","Type":"ContainerDied","Data":"f4a02b0be6984ccc1bb58c5b268bfde35355fca4c40603dfac6f57f3cbb7442b"}
Sep 19 10:57:10.078695 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.078595    2580 scope.go:117] "RemoveContainer" containerID="2d2aa346e190da13a7edbaa3f85f669b42ab61c71bf4c5cc9f6918af5d07ba88"
Sep 19 10:57:10.115238 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.115202    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:57:10.205939 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.205907    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:10.241218 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241188    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-workload-socket\") pod \"caa42699-c4cd-41a8-8825-61cb1b057d42\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") "
Sep 19 10:57:10.241408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsxtt\" (UniqueName: \"kubernetes.io/projected/caa42699-c4cd-41a8-8825-61cb1b057d42-kube-api-access-zsxtt\") pod \"caa42699-c4cd-41a8-8825-61cb1b057d42\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") "
Sep 19 10:57:10.241408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241265    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-data\") pod \"caa42699-c4cd-41a8-8825-61cb1b057d42\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") "
Sep 19 10:57:10.241408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241282    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-podinfo\") pod \"caa42699-c4cd-41a8-8825-61cb1b057d42\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") "
Sep 19 10:57:10.241408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241317    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-workload-certs\") pod \"caa42699-c4cd-41a8-8825-61cb1b057d42\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") "
Sep 19 10:57:10.241408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241335    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa42699-c4cd-41a8-8825-61cb1b057d42-istiod-ca-cert\") pod \"caa42699-c4cd-41a8-8825-61cb1b057d42\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") "
Sep 19 10:57:10.241408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-credential-socket\") pod \"caa42699-c4cd-41a8-8825-61cb1b057d42\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") "
Sep 19 10:57:10.241408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241399    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-token\") pod \"caa42699-c4cd-41a8-8825-61cb1b057d42\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") "
Sep 19 10:57:10.241790 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241441    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-envoy\") pod \"caa42699-c4cd-41a8-8825-61cb1b057d42\" (UID: \"caa42699-c4cd-41a8-8825-61cb1b057d42\") "
Sep 19 10:57:10.241790 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "caa42699-c4cd-41a8-8825-61cb1b057d42" (UID: "caa42699-c4cd-41a8-8825-61cb1b057d42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:57:10.241790 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241714    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "caa42699-c4cd-41a8-8825-61cb1b057d42" (UID: "caa42699-c4cd-41a8-8825-61cb1b057d42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:57:10.241949 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241847    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-workload-socket\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:57:10.241949 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241870    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-workload-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:57:10.241949 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241845    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "caa42699-c4cd-41a8-8825-61cb1b057d42" (UID: "caa42699-c4cd-41a8-8825-61cb1b057d42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:57:10.241949 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.241899    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-data" (OuterVolumeSpecName: "istio-data") pod "caa42699-c4cd-41a8-8825-61cb1b057d42" (UID: "caa42699-c4cd-41a8-8825-61cb1b057d42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:57:10.242089 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.242034    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa42699-c4cd-41a8-8825-61cb1b057d42-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "caa42699-c4cd-41a8-8825-61cb1b057d42" (UID: "caa42699-c4cd-41a8-8825-61cb1b057d42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:57:10.244320 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.244296    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa42699-c4cd-41a8-8825-61cb1b057d42-kube-api-access-zsxtt" (OuterVolumeSpecName: "kube-api-access-zsxtt") pod "caa42699-c4cd-41a8-8825-61cb1b057d42" (UID: "caa42699-c4cd-41a8-8825-61cb1b057d42"). InnerVolumeSpecName "kube-api-access-zsxtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:57:10.244441 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.244328    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-token" (OuterVolumeSpecName: "istio-token") pod "caa42699-c4cd-41a8-8825-61cb1b057d42" (UID: "caa42699-c4cd-41a8-8825-61cb1b057d42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:57:10.244441 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.244375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "caa42699-c4cd-41a8-8825-61cb1b057d42" (UID: "caa42699-c4cd-41a8-8825-61cb1b057d42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 10:57:10.244441 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.244402    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "caa42699-c4cd-41a8-8825-61cb1b057d42" (UID: "caa42699-c4cd-41a8-8825-61cb1b057d42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:57:10.342585 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.342549    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-envoy\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:57:10.342585 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.342580    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsxtt\" (UniqueName: \"kubernetes.io/projected/caa42699-c4cd-41a8-8825-61cb1b057d42-kube-api-access-zsxtt\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:57:10.342585 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.342591    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-data\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:57:10.342882 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.342599    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-podinfo\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:57:10.342882 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.342607    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa42699-c4cd-41a8-8825-61cb1b057d42-istiod-ca-cert\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:57:10.342882 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.342615    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa42699-c4cd-41a8-8825-61cb1b057d42-credential-socket\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:57:10.342882 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.342622    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa42699-c4cd-41a8-8825-61cb1b057d42-istio-token\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:57:10.841416 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:10.841380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:11.085603 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:11.085570    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"
Sep 19 10:57:11.085603 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:11.085593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8" event={"ID":"caa42699-c4cd-41a8-8825-61cb1b057d42","Type":"ContainerDied","Data":"a703826068f5a746199687d9892fb1fbf08041120ce642a8be0509bb4064ffac"}
Sep 19 10:57:11.085879 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:11.085654    2580 scope.go:117] "RemoveContainer" containerID="f4a02b0be6984ccc1bb58c5b268bfde35355fca4c40603dfac6f57f3cbb7442b"
Sep 19 10:57:11.111882 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:11.111846    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"]
Sep 19 10:57:11.115330 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:11.115297    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-df95e5-openshift-default-6d94f96996-spdz8"]
Sep 19 10:57:11.586590 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:11.586548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:11.857026 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:11.856944    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" path="/var/lib/kubelet/pods/caa42699-c4cd-41a8-8825-61cb1b057d42/volumes"
Sep 19 10:57:12.326246 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.326212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:12.438459 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.438423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:12.577957 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.577861    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:12.819476 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.819442    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:12.884963 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.884853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:12.950644 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.950607    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"]
Sep 19 10:57:12.951032 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951019    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerName="istio-proxy"
Sep 19 10:57:12.951086 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951035    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerName="istio-proxy"
Sep 19 10:57:12.951086 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951047    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2017702a-00f7-487f-babd-0057c8fdcbed" containerName="maas-api"
Sep 19 10:57:12.951086 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951053    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2017702a-00f7-487f-babd-0057c8fdcbed" containerName="maas-api"
Sep 19 10:57:12.951086 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951059    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerName="istio-proxy"
Sep 19 10:57:12.951086 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951064    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerName="istio-proxy"
Sep 19 10:57:12.951086 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951070    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3c8428f-76a5-423a-a85a-8a94afd89cf9" containerName="main"
Sep 19 10:57:12.951086 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951075    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3c8428f-76a5-423a-a85a-8a94afd89cf9" containerName="main"
Sep 19 10:57:12.951086 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951089    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd962b8a-8a90-42ea-8d4b-9295601dafd8" containerName="main"
Sep 19 10:57:12.951326 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951094    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd962b8a-8a90-42ea-8d4b-9295601dafd8" containerName="main"
Sep 19 10:57:12.951326 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951170    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerName="istio-proxy"
Sep 19 10:57:12.951326 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951177    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2017702a-00f7-487f-babd-0057c8fdcbed" containerName="maas-api"
Sep 19 10:57:12.951326 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951185    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd962b8a-8a90-42ea-8d4b-9295601dafd8" containerName="main"
Sep 19 10:57:12.951326 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.951192    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3c8428f-76a5-423a-a85a-8a94afd89cf9" containerName="main"
Sep 19 10:57:12.955705 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.955686    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"
Sep 19 10:57:12.957808 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.957745    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-137f88-dockercfg-rwdf2\""
Sep 19 10:57:12.957952 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.957911    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-137f88\""
Sep 19 10:57:12.964115 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:12.964081    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"]
Sep 19 10:57:13.069296 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:13.069254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf964\" (UniqueName: \"kubernetes.io/projected/4d5432bb-551f-4efa-8234-686269667977-kube-api-access-cf964\") pod \"payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk\" (UID: \"4d5432bb-551f-4efa-8234-686269667977\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"
Sep 19 10:57:13.069490 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:13.069304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5432bb-551f-4efa-8234-686269667977-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk\" (UID: \"4d5432bb-551f-4efa-8234-686269667977\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"
Sep 19 10:57:13.170150 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:13.170054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf964\" (UniqueName: \"kubernetes.io/projected/4d5432bb-551f-4efa-8234-686269667977-kube-api-access-cf964\") pod \"payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk\" (UID: \"4d5432bb-551f-4efa-8234-686269667977\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"
Sep 19 10:57:13.170150 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:13.170101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5432bb-551f-4efa-8234-686269667977-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk\" (UID: \"4d5432bb-551f-4efa-8234-686269667977\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"
Sep 19 10:57:13.170808 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:13.170789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5432bb-551f-4efa-8234-686269667977-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk\" (UID: \"4d5432bb-551f-4efa-8234-686269667977\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"
Sep 19 10:57:13.178203 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:13.178181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf964\" (UniqueName: \"kubernetes.io/projected/4d5432bb-551f-4efa-8234-686269667977-kube-api-access-cf964\") pod \"payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk\" (UID: \"4d5432bb-551f-4efa-8234-686269667977\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"
Sep 19 10:57:13.269313 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:13.269265    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"
Sep 19 10:57:13.543330 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:13.543303    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"]
Sep 19 10:57:13.545345 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:57:13.545317    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5432bb_551f_4efa_8234_686269667977.slice/crio-dcda87da6c44c017736e61c6e3c33cf51dbd638acd01e86e376652da994bfba3 WatchSource:0}: Error finding container dcda87da6c44c017736e61c6e3c33cf51dbd638acd01e86e376652da994bfba3: Status 404 returned error can't find the container with id dcda87da6c44c017736e61c6e3c33cf51dbd638acd01e86e376652da994bfba3
Sep 19 10:57:14.103966 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:14.103931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk" event={"ID":"4d5432bb-551f-4efa-8234-686269667977","Type":"ContainerStarted","Data":"4cfad6b28c305d28feca8152d2e85b2aa3ef046956d7f41dec45cf676a4e4f4d"}
Sep 19 10:57:14.104390 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:14.103981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk" event={"ID":"4d5432bb-551f-4efa-8234-686269667977","Type":"ContainerStarted","Data":"dcda87da6c44c017736e61c6e3c33cf51dbd638acd01e86e376652da994bfba3"}
Sep 19 10:57:14.104390 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:14.104129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"
Sep 19 10:57:14.120005 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:14.119954    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk" podStartSLOduration=2.119938023 podStartE2EDuration="2.119938023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:57:14.118540233 +0000 UTC m=+1278.777285390" watchObservedRunningTime="2026-09-19 10:57:14.119938023 +0000 UTC m=+1278.778683183"
Sep 19 10:57:21.215529 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:21.215486    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:57:25.111850 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:25.111815    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"
Sep 19 10:57:28.812366 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:28.812326    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"]
Sep 19 10:57:28.813007 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:28.812989    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="caa42699-c4cd-41a8-8825-61cb1b057d42" containerName="istio-proxy"
Sep 19 10:57:28.816124 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:28.816100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:28.818282 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:28.818250    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1907c1\"/\"distinct-2-w0-1907c1-kserve-self-signed-certs\""
Sep 19 10:57:28.824405 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:28.824351    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"]
Sep 19 10:57:28.839735 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:28.839702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:28.925818 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:28.925778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-model-cache\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:28.926004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:28.925832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-home\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:28.926004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:28.925930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-tmp-dir\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:28.926004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:28.925970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7bqq\" (UniqueName: \"kubernetes.io/projected/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-kube-api-access-m7bqq\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:28.926110 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:28.926040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-tls-certs\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:28.926110 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:28.926057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-dshm\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.027030 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.026997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-model-cache\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.027218 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.027042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-home\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.027218 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.027088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-tmp-dir\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.027218 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.027108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7bqq\" (UniqueName: \"kubernetes.io/projected/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-kube-api-access-m7bqq\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.027218 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.027168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-tls-certs\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.027218 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.027191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-dshm\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.027483 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.027398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-model-cache\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.027483 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.027475    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-home\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.027603 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.027498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-tmp-dir\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.029640 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.029618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-dshm\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.029866 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.029849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-tls-certs\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.034996 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.034978    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7bqq\" (UniqueName: \"kubernetes.io/projected/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-kube-api-access-m7bqq\") pod \"distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") " pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.126474 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.126385    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:29.397580 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:29.397536    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"]
Sep 19 10:57:29.399434 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:57:29.399408    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e0a60ed_75c3_4c9c_a759_5bc9b46ef45c.slice/crio-84f35837a3bb6bc064b28a926dd0fdac86b7dec54e7f23e6d7828f1018bd55e2 WatchSource:0}: Error finding container 84f35837a3bb6bc064b28a926dd0fdac86b7dec54e7f23e6d7828f1018bd55e2: Status 404 returned error can't find the container with id 84f35837a3bb6bc064b28a926dd0fdac86b7dec54e7f23e6d7828f1018bd55e2
Sep 19 10:57:30.191656 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:30.191613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9" event={"ID":"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c","Type":"ContainerStarted","Data":"16a1b1df425f77da55a28e7dca8ed64d62013386ab417ffbc50b7f51a128ef08"}
Sep 19 10:57:30.192072 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:30.191666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9" event={"ID":"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c","Type":"ContainerStarted","Data":"84f35837a3bb6bc064b28a926dd0fdac86b7dec54e7f23e6d7828f1018bd55e2"}
Sep 19 10:57:30.208084 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:30.208027    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9" podStartSLOduration=2.208009853 podStartE2EDuration="2.208009853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:57:30.205131255 +0000 UTC m=+1294.863876416" watchObservedRunningTime="2026-09-19 10:57:30.208009853 +0000 UTC m=+1294.866755012"
Sep 19 10:57:33.189874 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:33.189831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:35.410700 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:35.410660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:36.129181 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:36.129146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:36.922037 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:36.922005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:37.726580 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:37.726543    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:38.557526 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:38.557492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:39.069816 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:39.069778    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:39.127045 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:39.127000    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:39.127231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:39.127107    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:39.139726 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:39.139696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:39.251914 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:39.251887    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 10:57:39.328590 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:39.328492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:39.429407 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:39.429374    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:39.681730 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:39.681697    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:40.067748 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:40.067654    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:40.165350 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:40.165312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:42.506445 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:42.506405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:43.068819 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:43.068785    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:43.164892 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:43.164859    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:43.268153 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:43.268118    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:43.924773 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:43.924727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:44.637679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:44.637647    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:45.612738 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:45.612689    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:46.235711 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:46.235673    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:46.853967 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:46.853928    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:47.541707 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:47.541667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:47.637438 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:47.637406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:47.738172 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:47.738129    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:47.992887 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:47.992838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:48.095204 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.095166    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:48.134248 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.134210    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"]
Sep 19 10:57:48.138066 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.138048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"
Sep 19 10:57:48.140152 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.140128    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-9b3bef\""
Sep 19 10:57:48.140273 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.140168    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-9b3bef-dockercfg-f62k6\""
Sep 19 10:57:48.147059 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.147022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"]
Sep 19 10:57:48.192068 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.192024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjznw\" (UniqueName: \"kubernetes.io/projected/1c4607f2-ca97-44e1-92e9-d874d59b2e5e-kube-api-access-pjznw\") pod \"payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt\" (UID: \"1c4607f2-ca97-44e1-92e9-d874d59b2e5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"
Sep 19 10:57:48.192253 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.192113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4607f2-ca97-44e1-92e9-d874d59b2e5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt\" (UID: \"1c4607f2-ca97-44e1-92e9-d874d59b2e5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"
Sep 19 10:57:48.292844 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.292728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4607f2-ca97-44e1-92e9-d874d59b2e5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt\" (UID: \"1c4607f2-ca97-44e1-92e9-d874d59b2e5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"
Sep 19 10:57:48.292844 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.292808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjznw\" (UniqueName: \"kubernetes.io/projected/1c4607f2-ca97-44e1-92e9-d874d59b2e5e-kube-api-access-pjznw\") pod \"payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt\" (UID: \"1c4607f2-ca97-44e1-92e9-d874d59b2e5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"
Sep 19 10:57:48.293533 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.293507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4607f2-ca97-44e1-92e9-d874d59b2e5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt\" (UID: \"1c4607f2-ca97-44e1-92e9-d874d59b2e5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"
Sep 19 10:57:48.300107 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.300085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjznw\" (UniqueName: \"kubernetes.io/projected/1c4607f2-ca97-44e1-92e9-d874d59b2e5e-kube-api-access-pjznw\") pod \"payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt\" (UID: \"1c4607f2-ca97-44e1-92e9-d874d59b2e5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"
Sep 19 10:57:48.450004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.449966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"
Sep 19 10:57:48.724297 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:48.724268    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"]
Sep 19 10:57:48.726313 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:57:48.726287    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4607f2_ca97_44e1_92e9_d874d59b2e5e.slice/crio-1c21bf434e10076ec5c93ad1456cfd4ab3aeea983d69329994e1cf31bb6758f6 WatchSource:0}: Error finding container 1c21bf434e10076ec5c93ad1456cfd4ab3aeea983d69329994e1cf31bb6758f6: Status 404 returned error can't find the container with id 1c21bf434e10076ec5c93ad1456cfd4ab3aeea983d69329994e1cf31bb6758f6
Sep 19 10:57:49.198447 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:49.198413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:49.295862 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:49.295824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt" event={"ID":"1c4607f2-ca97-44e1-92e9-d874d59b2e5e","Type":"ContainerStarted","Data":"382737a9128eec382951bbfb0d5a07597d2b30e7202995686e54149049d03ffe"}
Sep 19 10:57:49.296038 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:49.295880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt" event={"ID":"1c4607f2-ca97-44e1-92e9-d874d59b2e5e","Type":"ContainerStarted","Data":"1c21bf434e10076ec5c93ad1456cfd4ab3aeea983d69329994e1cf31bb6758f6"}
Sep 19 10:57:49.296038 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:49.295959    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"
Sep 19 10:57:49.311808 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:49.311732    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt" podStartSLOduration=1.3117148410000001 podStartE2EDuration="1.311714841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:57:49.309642444 +0000 UTC m=+1313.968387603" watchObservedRunningTime="2026-09-19 10:57:49.311714841 +0000 UTC m=+1313.970460043"
Sep 19 10:57:49.924880 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:49.924842    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:50.640111 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:50.640065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:51.253964 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:51.253928    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:52.019137 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:52.019099    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:52.729296 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:52.729269    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:53.527976 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:53.527945    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:54.117384 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:54.117351    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:54.743039 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:54.743007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:55.404796 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:55.404739    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:57:55.723676 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:55.723640    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:57:58.196345 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:58.196310    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"]
Sep 19 10:57:58.215696 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:57:58.215659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:00.303405 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:00.303365    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"
Sep 19 10:58:03.965473 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:03.965416    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"]
Sep 19 10:58:03.966281 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:03.966239    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms" podUID="ae0f0d0f-c34c-4e01-b2c7-4e43700e2925" containerName="maas-api" containerID="cri-o://d4f9bd9d6a02fcdcf7958cb8618a7b3c7856669c7289532a363c3ba626ebae46" gracePeriod=30
Sep 19 10:58:03.994814 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:03.994784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:04.027261 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.027223    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"]
Sep 19 10:58:04.027510 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.027482    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr" podUID="9573261a-046e-4fd9-9953-37a9f2efeec6" containerName="payload-pre-processing" containerID="cri-o://f687250570362e9e681f8767cf8ec96ffa30dfca50f8cc5e57984eaadf9bdb0d" gracePeriod=30
Sep 19 10:58:04.092539 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.092502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:04.173251 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.173147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:04.222560 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.222430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:04.301408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.301380    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"
Sep 19 10:58:04.304176 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.304149    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:58:04.376673 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.376641    2580 generic.go:356] "Generic (PLEG): container finished" podID="ae0f0d0f-c34c-4e01-b2c7-4e43700e2925" containerID="d4f9bd9d6a02fcdcf7958cb8618a7b3c7856669c7289532a363c3ba626ebae46" exitCode=0
Sep 19 10:58:04.376877 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.376692    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"
Sep 19 10:58:04.376877 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.376709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms" event={"ID":"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925","Type":"ContainerDied","Data":"d4f9bd9d6a02fcdcf7958cb8618a7b3c7856669c7289532a363c3ba626ebae46"}
Sep 19 10:58:04.376877 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.376787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms" event={"ID":"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925","Type":"ContainerDied","Data":"2488281668fbcdbe855f57321df9609f28356aaee6d64318202d125eb608cf70"}
Sep 19 10:58:04.376877 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.376822    2580 scope.go:117] "RemoveContainer" containerID="d4f9bd9d6a02fcdcf7958cb8618a7b3c7856669c7289532a363c3ba626ebae46"
Sep 19 10:58:04.378289 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.378254    2580 generic.go:356] "Generic (PLEG): container finished" podID="9573261a-046e-4fd9-9953-37a9f2efeec6" containerID="f687250570362e9e681f8767cf8ec96ffa30dfca50f8cc5e57984eaadf9bdb0d" exitCode=0
Sep 19 10:58:04.378380 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.378303    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"
Sep 19 10:58:04.378380 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.378316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr" event={"ID":"9573261a-046e-4fd9-9953-37a9f2efeec6","Type":"ContainerDied","Data":"f687250570362e9e681f8767cf8ec96ffa30dfca50f8cc5e57984eaadf9bdb0d"}
Sep 19 10:58:04.378380 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.378355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr" event={"ID":"9573261a-046e-4fd9-9953-37a9f2efeec6","Type":"ContainerDied","Data":"d97f4bc1358f191fae771f54b76a1745aa34507b132f0fbb9d2021c46551c67b"}
Sep 19 10:58:04.386487 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.386471    2580 scope.go:117] "RemoveContainer" containerID="d4f9bd9d6a02fcdcf7958cb8618a7b3c7856669c7289532a363c3ba626ebae46"
Sep 19 10:58:04.386836 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:58:04.386816    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f9bd9d6a02fcdcf7958cb8618a7b3c7856669c7289532a363c3ba626ebae46\": container with ID starting with d4f9bd9d6a02fcdcf7958cb8618a7b3c7856669c7289532a363c3ba626ebae46 not found: ID does not exist" containerID="d4f9bd9d6a02fcdcf7958cb8618a7b3c7856669c7289532a363c3ba626ebae46"
Sep 19 10:58:04.386897 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.386845    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f9bd9d6a02fcdcf7958cb8618a7b3c7856669c7289532a363c3ba626ebae46"} err="failed to get container status \"d4f9bd9d6a02fcdcf7958cb8618a7b3c7856669c7289532a363c3ba626ebae46\": rpc error: code = NotFound desc = could not find container \"d4f9bd9d6a02fcdcf7958cb8618a7b3c7856669c7289532a363c3ba626ebae46\": container with ID starting with d4f9bd9d6a02fcdcf7958cb8618a7b3c7856669c7289532a363c3ba626ebae46 not found: ID does not exist"
Sep 19 10:58:04.386897 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.386863    2580 scope.go:117] "RemoveContainer" containerID="f687250570362e9e681f8767cf8ec96ffa30dfca50f8cc5e57984eaadf9bdb0d"
Sep 19 10:58:04.395417 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.395399    2580 scope.go:117] "RemoveContainer" containerID="f687250570362e9e681f8767cf8ec96ffa30dfca50f8cc5e57984eaadf9bdb0d"
Sep 19 10:58:04.395714 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:58:04.395694    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f687250570362e9e681f8767cf8ec96ffa30dfca50f8cc5e57984eaadf9bdb0d\": container with ID starting with f687250570362e9e681f8767cf8ec96ffa30dfca50f8cc5e57984eaadf9bdb0d not found: ID does not exist" containerID="f687250570362e9e681f8767cf8ec96ffa30dfca50f8cc5e57984eaadf9bdb0d"
Sep 19 10:58:04.395792 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.395724    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f687250570362e9e681f8767cf8ec96ffa30dfca50f8cc5e57984eaadf9bdb0d"} err="failed to get container status \"f687250570362e9e681f8767cf8ec96ffa30dfca50f8cc5e57984eaadf9bdb0d\": rpc error: code = NotFound desc = could not find container \"f687250570362e9e681f8767cf8ec96ffa30dfca50f8cc5e57984eaadf9bdb0d\": container with ID starting with f687250570362e9e681f8767cf8ec96ffa30dfca50f8cc5e57984eaadf9bdb0d not found: ID does not exist"
Sep 19 10:58:04.453606 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.453570    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-metrics-certs\") pod \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") "
Sep 19 10:58:04.453816 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.453635    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-openshift-service-ca\") pod \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") "
Sep 19 10:58:04.453816 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.453687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-maas-api-tls\") pod \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") "
Sep 19 10:58:04.453816 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.453728    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9573261a-046e-4fd9-9953-37a9f2efeec6-plugins-config-volume\") pod \"9573261a-046e-4fd9-9953-37a9f2efeec6\" (UID: \"9573261a-046e-4fd9-9953-37a9f2efeec6\") "
Sep 19 10:58:04.453988 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.453822    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6bdf\" (UniqueName: \"kubernetes.io/projected/9573261a-046e-4fd9-9953-37a9f2efeec6-kube-api-access-b6bdf\") pod \"9573261a-046e-4fd9-9953-37a9f2efeec6\" (UID: \"9573261a-046e-4fd9-9953-37a9f2efeec6\") "
Sep 19 10:58:04.453988 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.453864    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-689tw\" (UniqueName: \"kubernetes.io/projected/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-kube-api-access-689tw\") pod \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\" (UID: \"ae0f0d0f-c34c-4e01-b2c7-4e43700e2925\") "
Sep 19 10:58:04.454094 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.454067    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae0f0d0f-c34c-4e01-b2c7-4e43700e2925" (UID: "ae0f0d0f-c34c-4e01-b2c7-4e43700e2925"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:58:04.454208 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.454192    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-openshift-service-ca\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:04.454293 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.454196    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9573261a-046e-4fd9-9953-37a9f2efeec6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9573261a-046e-4fd9-9953-37a9f2efeec6" (UID: "9573261a-046e-4fd9-9953-37a9f2efeec6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 10:58:04.456237 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.456214    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9573261a-046e-4fd9-9953-37a9f2efeec6-kube-api-access-b6bdf" (OuterVolumeSpecName: "kube-api-access-b6bdf") pod "9573261a-046e-4fd9-9953-37a9f2efeec6" (UID: "9573261a-046e-4fd9-9953-37a9f2efeec6"). InnerVolumeSpecName "kube-api-access-b6bdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:58:04.456405 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.456386    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae0f0d0f-c34c-4e01-b2c7-4e43700e2925" (UID: "ae0f0d0f-c34c-4e01-b2c7-4e43700e2925"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:58:04.456606 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.456585    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ae0f0d0f-c34c-4e01-b2c7-4e43700e2925" (UID: "ae0f0d0f-c34c-4e01-b2c7-4e43700e2925"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:58:04.456606 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.456589    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-kube-api-access-689tw" (OuterVolumeSpecName: "kube-api-access-689tw") pod "ae0f0d0f-c34c-4e01-b2c7-4e43700e2925" (UID: "ae0f0d0f-c34c-4e01-b2c7-4e43700e2925"). InnerVolumeSpecName "kube-api-access-689tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:58:04.555566 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.555473    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9573261a-046e-4fd9-9953-37a9f2efeec6-plugins-config-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:04.555566 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.555503    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6bdf\" (UniqueName: \"kubernetes.io/projected/9573261a-046e-4fd9-9953-37a9f2efeec6-kube-api-access-b6bdf\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:04.555566 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.555515    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-689tw\" (UniqueName: \"kubernetes.io/projected/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-kube-api-access-689tw\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:04.555566 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.555524    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-metrics-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:04.555566 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.555533    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925-maas-api-tls\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:04.599117 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.599079    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:04.701197 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.701168    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"]
Sep 19 10:58:04.705148 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.705118    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9ebfd604-bdfb59c74-4gnqr"]
Sep 19 10:58:04.717096 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.717068    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"]
Sep 19 10:58:04.718799 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:04.718780    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9ebfd604-7fc598d6dd-fg9ms"]
Sep 19 10:58:05.417004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:05.416964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:05.857787 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:05.857736    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9573261a-046e-4fd9-9953-37a9f2efeec6" path="/var/lib/kubelet/pods/9573261a-046e-4fd9-9953-37a9f2efeec6/volumes"
Sep 19 10:58:05.858311 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:05.858289    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae0f0d0f-c34c-4e01-b2c7-4e43700e2925" path="/var/lib/kubelet/pods/ae0f0d0f-c34c-4e01-b2c7-4e43700e2925/volumes"
Sep 19 10:58:06.141167 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:06.141065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:07.075513 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:07.075473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:07.244551 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:07.244517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:07.732519 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:07.732485    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:08.008488 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:08.008405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:08.615797 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:08.615747    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:09.326501 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:09.326459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:10.011838 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.011796    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"]
Sep 19 10:58:10.012261 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.012177    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9573261a-046e-4fd9-9953-37a9f2efeec6" containerName="payload-pre-processing"
Sep 19 10:58:10.012261 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.012189    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9573261a-046e-4fd9-9953-37a9f2efeec6" containerName="payload-pre-processing"
Sep 19 10:58:10.012261 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.012209    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae0f0d0f-c34c-4e01-b2c7-4e43700e2925" containerName="maas-api"
Sep 19 10:58:10.012261 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.012214    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae0f0d0f-c34c-4e01-b2c7-4e43700e2925" containerName="maas-api"
Sep 19 10:58:10.012417 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.012284    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae0f0d0f-c34c-4e01-b2c7-4e43700e2925" containerName="maas-api"
Sep 19 10:58:10.012417 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.012294    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9573261a-046e-4fd9-9953-37a9f2efeec6" containerName="payload-pre-processing"
Sep 19 10:58:10.016811 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.016790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.019494 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.019469    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5116ec\"/\"distinct-w4-5116ec-kserve-self-signed-certs\""
Sep 19 10:58:10.026861 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.026836    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"]
Sep 19 10:58:10.034123 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.034093    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:10.112867 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.112831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-tls-certs\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.113079 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.112881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-tmp-dir\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.113079 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.112969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-dshm\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.113079 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.112992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljgw5\" (UniqueName: \"kubernetes.io/projected/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-kube-api-access-ljgw5\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.113079 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.113043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-home\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.113252 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.113101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-model-cache\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.136338 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.136303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:10.214579 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.214542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-dshm\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.214579 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.214579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljgw5\" (UniqueName: \"kubernetes.io/projected/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-kube-api-access-ljgw5\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.214909 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.214600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-home\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.214909 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.214786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-model-cache\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.214909 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.214847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-tls-certs\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.214909 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.214878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-tmp-dir\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.215143 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.215116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-model-cache\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.215186 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.215111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-home\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.215224 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.215182    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-tmp-dir\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.217016 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.216998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-dshm\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.217434 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.217416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-tls-certs\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.221803 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.221781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljgw5\" (UniqueName: \"kubernetes.io/projected/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-kube-api-access-ljgw5\") pod \"distinct-w4-5116ec-kserve-dcb7456f8-z6knr\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") " pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.327026 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.326926    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:10.600064 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:10.600036    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"]
Sep 19 10:58:10.602387 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:58:10.602361    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb65cc2f_1fda_4c77_b076_7e4dd789bfdf.slice/crio-13848fbcbb968ee21ff1697cdea5fda44b406d9d97504d69e96c165e15b9cfb7 WatchSource:0}: Error finding container 13848fbcbb968ee21ff1697cdea5fda44b406d9d97504d69e96c165e15b9cfb7: Status 404 returned error can't find the container with id 13848fbcbb968ee21ff1697cdea5fda44b406d9d97504d69e96c165e15b9cfb7
Sep 19 10:58:11.030694 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:11.030657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:11.418669 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:11.418575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr" event={"ID":"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf","Type":"ContainerStarted","Data":"70044462fd73d3060fd7c084663a10bff25d3b3f2609584449b3f5bb53b216c1"}
Sep 19 10:58:11.418669 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:11.418625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr" event={"ID":"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf","Type":"ContainerStarted","Data":"13848fbcbb968ee21ff1697cdea5fda44b406d9d97504d69e96c165e15b9cfb7"}
Sep 19 10:58:11.435090 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:11.435032    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr" podStartSLOduration=2.43501803 podStartE2EDuration="2.43501803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:58:11.434172835 +0000 UTC m=+1336.092917994" watchObservedRunningTime="2026-09-19 10:58:11.43501803 +0000 UTC m=+1336.093763226"
Sep 19 10:58:11.723604 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:11.723564    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:12.225071 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:12.225030    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"]
Sep 19 10:58:12.321093 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:12.321054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:12.422926 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:12.422886    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:13.216535 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.216489    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn" podUID="fa3ff5bf-d397-4760-9ee3-e235b143e6a5" containerName="main" containerID="cri-o://111e7f526963e6a7b712e39567b5b85cb0b7f36f9c39d10493b4c51d37e48d72" gracePeriod=45
Sep 19 10:58:13.432594 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.432561    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa3ff5bf-d397-4760-9ee3-e235b143e6a5" containerID="111e7f526963e6a7b712e39567b5b85cb0b7f36f9c39d10493b4c51d37e48d72" exitCode=0
Sep 19 10:58:13.432968 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.432605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn" event={"ID":"fa3ff5bf-d397-4760-9ee3-e235b143e6a5","Type":"ContainerDied","Data":"111e7f526963e6a7b712e39567b5b85cb0b7f36f9c39d10493b4c51d37e48d72"}
Sep 19 10:58:13.487342 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.487316    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:58:13.654182 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.654142    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-tmp-dir\") pod \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") "
Sep 19 10:58:13.654380 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.654194    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-dshm\") pod \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") "
Sep 19 10:58:13.654380 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.654245    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-tls-certs\") pod \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") "
Sep 19 10:58:13.654380 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.654265    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhzxn\" (UniqueName: \"kubernetes.io/projected/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-kube-api-access-fhzxn\") pod \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") "
Sep 19 10:58:13.654380 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.654340    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-home\") pod \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") "
Sep 19 10:58:13.654610 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.654405    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-model-cache\") pod \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\" (UID: \"fa3ff5bf-d397-4760-9ee3-e235b143e6a5\") "
Sep 19 10:58:13.654610 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.654478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa3ff5bf-d397-4760-9ee3-e235b143e6a5" (UID: "fa3ff5bf-d397-4760-9ee3-e235b143e6a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:58:13.654715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.654624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-home" (OuterVolumeSpecName: "home") pod "fa3ff5bf-d397-4760-9ee3-e235b143e6a5" (UID: "fa3ff5bf-d397-4760-9ee3-e235b143e6a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:58:13.654715 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.654657    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa3ff5bf-d397-4760-9ee3-e235b143e6a5" (UID: "fa3ff5bf-d397-4760-9ee3-e235b143e6a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:58:13.654838 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.654796    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:13.654838 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.654809    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:13.654838 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.654819    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:13.656714 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.656685    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa3ff5bf-d397-4760-9ee3-e235b143e6a5" (UID: "fa3ff5bf-d397-4760-9ee3-e235b143e6a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:58:13.656938 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.656914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-kube-api-access-fhzxn" (OuterVolumeSpecName: "kube-api-access-fhzxn") pod "fa3ff5bf-d397-4760-9ee3-e235b143e6a5" (UID: "fa3ff5bf-d397-4760-9ee3-e235b143e6a5"). InnerVolumeSpecName "kube-api-access-fhzxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:58:13.657004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.656969    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-dshm" (OuterVolumeSpecName: "dshm") pod "fa3ff5bf-d397-4760-9ee3-e235b143e6a5" (UID: "fa3ff5bf-d397-4760-9ee3-e235b143e6a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:58:13.756257 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.756149    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:13.756257 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.756199    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhzxn\" (UniqueName: \"kubernetes.io/projected/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-kube-api-access-fhzxn\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:13.756257 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:13.756210    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3ff5bf-d397-4760-9ee3-e235b143e6a5-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:14.439332 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:14.439303    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"
Sep 19 10:58:14.439798 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:14.439307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn" event={"ID":"fa3ff5bf-d397-4760-9ee3-e235b143e6a5","Type":"ContainerDied","Data":"35dfd927302ab6b53afd1a036425cf6e6cfafe290202156432d3c84702991816"}
Sep 19 10:58:14.439798 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:14.439429    2580 scope.go:117] "RemoveContainer" containerID="111e7f526963e6a7b712e39567b5b85cb0b7f36f9c39d10493b4c51d37e48d72"
Sep 19 10:58:14.455002 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:14.454975    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"]
Sep 19 10:58:14.458555 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:14.458520    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/e2e-conflict-a-9f7696ca-kserve-757fb8b8c4-9m9qn"]
Sep 19 10:58:15.858207 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:15.858171    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa3ff5bf-d397-4760-9ee3-e235b143e6a5" path="/var/lib/kubelet/pods/fa3ff5bf-d397-4760-9ee3-e235b143e6a5/volumes"
Sep 19 10:58:20.328003 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:20.327897    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:20.328003 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:20.327978    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:20.340676 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:20.340651    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:20.490867 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:20.490838    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 10:58:27.245148 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.245083    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn" podUID="6bee274b-77b5-40c8-b213-bcf52c2eadbb" containerName="main" containerID="cri-o://7951a74fea23b38d87d1ee6d5a45c62f08fa95469d723e8557d9b1dd4391d4b0" gracePeriod=45
Sep 19 10:58:27.501272 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.501200    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:58:27.514693 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.514662    2580 generic.go:356] "Generic (PLEG): container finished" podID="6bee274b-77b5-40c8-b213-bcf52c2eadbb" containerID="7951a74fea23b38d87d1ee6d5a45c62f08fa95469d723e8557d9b1dd4391d4b0" exitCode=0
Sep 19 10:58:27.514899 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.514709    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"
Sep 19 10:58:27.514899 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.514736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn" event={"ID":"6bee274b-77b5-40c8-b213-bcf52c2eadbb","Type":"ContainerDied","Data":"7951a74fea23b38d87d1ee6d5a45c62f08fa95469d723e8557d9b1dd4391d4b0"}
Sep 19 10:58:27.514899 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.514803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn" event={"ID":"6bee274b-77b5-40c8-b213-bcf52c2eadbb","Type":"ContainerDied","Data":"e87b79230aa9feae79a6f82a40cd21e5993042281dd6dcc92f9f79679fc5da91"}
Sep 19 10:58:27.514899 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.514827    2580 scope.go:117] "RemoveContainer" containerID="7951a74fea23b38d87d1ee6d5a45c62f08fa95469d723e8557d9b1dd4391d4b0"
Sep 19 10:58:27.527113 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.527082    2580 scope.go:117] "RemoveContainer" containerID="7951a74fea23b38d87d1ee6d5a45c62f08fa95469d723e8557d9b1dd4391d4b0"
Sep 19 10:58:27.527547 ip-10-0-129-220 kubenswrapper[2580]: E0919 10:58:27.527509    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7951a74fea23b38d87d1ee6d5a45c62f08fa95469d723e8557d9b1dd4391d4b0\": container with ID starting with 7951a74fea23b38d87d1ee6d5a45c62f08fa95469d723e8557d9b1dd4391d4b0 not found: ID does not exist" containerID="7951a74fea23b38d87d1ee6d5a45c62f08fa95469d723e8557d9b1dd4391d4b0"
Sep 19 10:58:27.527642 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.527555    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7951a74fea23b38d87d1ee6d5a45c62f08fa95469d723e8557d9b1dd4391d4b0"} err="failed to get container status \"7951a74fea23b38d87d1ee6d5a45c62f08fa95469d723e8557d9b1dd4391d4b0\": rpc error: code = NotFound desc = could not find container \"7951a74fea23b38d87d1ee6d5a45c62f08fa95469d723e8557d9b1dd4391d4b0\": container with ID starting with 7951a74fea23b38d87d1ee6d5a45c62f08fa95469d723e8557d9b1dd4391d4b0 not found: ID does not exist"
Sep 19 10:58:27.584062 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.584032    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-home\") pod \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") "
Sep 19 10:58:27.584062 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.584073    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-tmp-dir\") pod \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") "
Sep 19 10:58:27.584325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.584124    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bee274b-77b5-40c8-b213-bcf52c2eadbb-tls-certs\") pod \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") "
Sep 19 10:58:27.584325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.584176    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxszg\" (UniqueName: \"kubernetes.io/projected/6bee274b-77b5-40c8-b213-bcf52c2eadbb-kube-api-access-gxszg\") pod \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") "
Sep 19 10:58:27.584325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.584246    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-dshm\") pod \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") "
Sep 19 10:58:27.584325 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.584272    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-model-cache\") pod \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\" (UID: \"6bee274b-77b5-40c8-b213-bcf52c2eadbb\") "
Sep 19 10:58:27.584508 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.584350    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-home" (OuterVolumeSpecName: "home") pod "6bee274b-77b5-40c8-b213-bcf52c2eadbb" (UID: "6bee274b-77b5-40c8-b213-bcf52c2eadbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:58:27.584508 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.584393    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6bee274b-77b5-40c8-b213-bcf52c2eadbb" (UID: "6bee274b-77b5-40c8-b213-bcf52c2eadbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:58:27.584614 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.584593    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-model-cache" (OuterVolumeSpecName: "model-cache") pod "6bee274b-77b5-40c8-b213-bcf52c2eadbb" (UID: "6bee274b-77b5-40c8-b213-bcf52c2eadbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:58:27.584675 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.584629    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:27.584675 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.584648    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:27.586640 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.586606    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-dshm" (OuterVolumeSpecName: "dshm") pod "6bee274b-77b5-40c8-b213-bcf52c2eadbb" (UID: "6bee274b-77b5-40c8-b213-bcf52c2eadbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 10:58:27.586640 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.586613    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bee274b-77b5-40c8-b213-bcf52c2eadbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bee274b-77b5-40c8-b213-bcf52c2eadbb" (UID: "6bee274b-77b5-40c8-b213-bcf52c2eadbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 10:58:27.586857 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.586735    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bee274b-77b5-40c8-b213-bcf52c2eadbb-kube-api-access-gxszg" (OuterVolumeSpecName: "kube-api-access-gxszg") pod "6bee274b-77b5-40c8-b213-bcf52c2eadbb" (UID: "6bee274b-77b5-40c8-b213-bcf52c2eadbb"). InnerVolumeSpecName "kube-api-access-gxszg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 10:58:27.686004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.685965    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:27.686004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.685999    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bee274b-77b5-40c8-b213-bcf52c2eadbb-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:27.686004 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.686008    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bee274b-77b5-40c8-b213-bcf52c2eadbb-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:27.686253 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.686017    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxszg\" (UniqueName: \"kubernetes.io/projected/6bee274b-77b5-40c8-b213-bcf52c2eadbb-kube-api-access-gxszg\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 10:58:27.834646 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.834615    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"]
Sep 19 10:58:27.838209 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.838187    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9ebfd604/e2e-lifecycle-model-9ebfd604-kserve-d7d4744b7-q4mwn"]
Sep 19 10:58:27.857932 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:27.857894    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bee274b-77b5-40c8-b213-bcf52c2eadbb" path="/var/lib/kubelet/pods/6bee274b-77b5-40c8-b213-bcf52c2eadbb/volumes"
Sep 19 10:58:34.400663 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.400623    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"]
Sep 19 10:58:34.401121 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.401015    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bee274b-77b5-40c8-b213-bcf52c2eadbb" containerName="main"
Sep 19 10:58:34.401121 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.401027    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bee274b-77b5-40c8-b213-bcf52c2eadbb" containerName="main"
Sep 19 10:58:34.401121 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.401042    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa3ff5bf-d397-4760-9ee3-e235b143e6a5" containerName="main"
Sep 19 10:58:34.401121 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.401048    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa3ff5bf-d397-4760-9ee3-e235b143e6a5" containerName="main"
Sep 19 10:58:34.401121 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.401111    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bee274b-77b5-40c8-b213-bcf52c2eadbb" containerName="main"
Sep 19 10:58:34.401121 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.401121    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa3ff5bf-d397-4760-9ee3-e235b143e6a5" containerName="main"
Sep 19 10:58:34.405473 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.405452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.407629 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.407602    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-137f88\"/\"default-dockercfg-986l4\""
Sep 19 10:58:34.408450 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.408427    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-137f88\"/\"auth-test-model-137f88-kserve-self-signed-certs\""
Sep 19 10:58:34.408578 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.408468    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-137f88\"/\"openshift-service-ca.crt\""
Sep 19 10:58:34.408578 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.408439    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-137f88\"/\"kube-root-ca.crt\""
Sep 19 10:58:34.413727 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.413702    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"]
Sep 19 10:58:34.432835 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.432791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:34.448865 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.448827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-model-cache\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.449032 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.448905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-dshm\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.449032 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.448941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-home\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.449032 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.448990    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-tmp-dir\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.449140 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.449050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-tls-certs\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.449140 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.449074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptb9f\" (UniqueName: \"kubernetes.io/projected/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-kube-api-access-ptb9f\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.549876 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.549831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-tls-certs\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.549876 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.549882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptb9f\" (UniqueName: \"kubernetes.io/projected/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-kube-api-access-ptb9f\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.550152 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.549918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-model-cache\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.550152 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.549981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-dshm\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.550152 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.550023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-home\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.550152 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.550063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-tmp-dir\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.550459 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.550429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-home\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.550586 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.550474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-tmp-dir\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.550586 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.550496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-model-cache\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.552377 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.552355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-dshm\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.552542 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.552525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-tls-certs\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.557169 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.557146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptb9f\" (UniqueName: \"kubernetes.io/projected/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-kube-api-access-ptb9f\") pod \"auth-test-model-137f88-kserve-cf66bc4bc-qkdd6\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") " pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.716087 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.716048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:34.988975 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:34.988949    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"]
Sep 19 10:58:34.990823 ip-10-0-129-220 kubenswrapper[2580]: W0919 10:58:34.990779    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cd5c909_ef48_41d3_85dc_c1b9bd0fd953.slice/crio-981877723cd282772aa0f0905a043f13c18fff379dc0fd960a97bd579fd37437 WatchSource:0}: Error finding container 981877723cd282772aa0f0905a043f13c18fff379dc0fd960a97bd579fd37437: Status 404 returned error can't find the container with id 981877723cd282772aa0f0905a043f13c18fff379dc0fd960a97bd579fd37437
Sep 19 10:58:35.570742 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:35.570709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6" event={"ID":"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953","Type":"ContainerStarted","Data":"e39589247cfb301883826c8cbb57127898ab72a5b8c071975eebdc602f90358d"}
Sep 19 10:58:35.570742 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:35.570746    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6" event={"ID":"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953","Type":"ContainerStarted","Data":"981877723cd282772aa0f0905a043f13c18fff379dc0fd960a97bd579fd37437"}
Sep 19 10:58:35.586518 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:35.586463    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6" podStartSLOduration=1.58644507 podStartE2EDuration="1.58644507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:58:35.585148571 +0000 UTC m=+1360.243893727" watchObservedRunningTime="2026-09-19 10:58:35.58644507 +0000 UTC m=+1360.245190230"
Sep 19 10:58:37.311945 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:37.311907    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:58:44.474012 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:44.473965    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:44.716959 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:44.716922    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:44.716959 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:44.716968    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:44.729916 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:44.729838    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:45.634196 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:45.634157    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 10:58:49.232867 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:49.232827    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:49.454308 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:49.454270    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:58:54.955608 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:58:54.955566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:00.322367 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:00.322330    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:59:05.724622 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:05.724589    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:06.552722 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:06.552676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:09.053378 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:09.053327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:09.152147 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:09.152112    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:09.306780 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:09.306666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:10.524730 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:10.524693    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:10.624635 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:10.624597    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:10.724863 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:10.724829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:11.326509 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:11.326473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:11.426726 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:11.426687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:11.527194 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:11.527161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:12.775324 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:12.775289    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:14.191813 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:14.191779    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:15.736530 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:15.736500    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:18.504095 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:18.504062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:18.604562 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:18.604519    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:18.668193 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:18.668156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:19.266038 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:19.265999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:24.956408 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:24.956364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:25.012748 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:25.012709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:25.081189 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:25.081146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:25.182207 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:25.182160    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:34.415386 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:34.415346    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 10:59:34.448679 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:34.448646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:35.636723 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:35.636684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:35.856231 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:35.856191    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:54.536521 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:54.536428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 10:59:56.768717 ip-10-0-129-220 kubenswrapper[2580]: I0919 10:59:56.768676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:00.137531 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.137495    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw"]
Sep 19 11:00:00.141824 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.141795    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29830260-wkdqz"]
Sep 19 11:00:00.141989 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.141911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw"
Sep 19 11:00:00.143983 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.143959    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r42s\""
Sep 19 11:00:00.144972 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.144957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-wkdqz"
Sep 19 11:00:00.151834 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.151806    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw"]
Sep 19 11:00:00.154466 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.154437    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29830260-wkdqz"]
Sep 19 11:00:00.155409 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.155387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw"
Sep 19 11:00:00.163286 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.160546    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb"]
Sep 19 11:00:00.175435 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.173319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-wkdqz"
Sep 19 11:00:00.175435 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.175003    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb"
Sep 19 11:00:00.175435 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.175407    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb"]
Sep 19 11:00:00.218526 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.218264    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb"
Sep 19 11:00:00.448612 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.448578    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw"]
Sep 19 11:00:00.451686 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:00:00.451662    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52b427c_5c69_49fc_a9b9_884cba258ec2.slice/crio-25c13095bf0d00c086f77abbd6029a9998d5c8c1de7670ae096e3b14c27d8973 WatchSource:0}: Error finding container 25c13095bf0d00c086f77abbd6029a9998d5c8c1de7670ae096e3b14c27d8973: Status 404 returned error can't find the container with id 25c13095bf0d00c086f77abbd6029a9998d5c8c1de7670ae096e3b14c27d8973
Sep 19 11:00:00.458958 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.458936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29830260-wkdqz"]
Sep 19 11:00:00.462156 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:00:00.462119    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dff9a57_bbb1_42fb_86fe_fd45d0ba32fb.slice/crio-1ffb460b7f3d506cc334d1958996b039ca0519d8f85a12bfa3457c709270e596 WatchSource:0}: Error finding container 1ffb460b7f3d506cc334d1958996b039ca0519d8f85a12bfa3457c709270e596: Status 404 returned error can't find the container with id 1ffb460b7f3d506cc334d1958996b039ca0519d8f85a12bfa3457c709270e596
Sep 19 11:00:00.521412 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:00.521380    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb"]
Sep 19 11:00:00.523288 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:00:00.523255    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f085bb5_808c_42d6_81da_cfdf6c8aa4b2.slice/crio-297200a5b15201a7e22c83f55a3d2fb25e572f015b9c5b5c3aa870cc19e1744f WatchSource:0}: Error finding container 297200a5b15201a7e22c83f55a3d2fb25e572f015b9c5b5c3aa870cc19e1744f: Status 404 returned error can't find the container with id 297200a5b15201a7e22c83f55a3d2fb25e572f015b9c5b5c3aa870cc19e1744f
Sep 19 11:00:01.022979 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:01.022941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb" event={"ID":"4f085bb5-808c-42d6-81da-cfdf6c8aa4b2","Type":"ContainerStarted","Data":"297200a5b15201a7e22c83f55a3d2fb25e572f015b9c5b5c3aa870cc19e1744f"}
Sep 19 11:00:01.024166 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:01.024134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw" event={"ID":"c52b427c-5c69-49fc-a9b9-884cba258ec2","Type":"ContainerStarted","Data":"25c13095bf0d00c086f77abbd6029a9998d5c8c1de7670ae096e3b14c27d8973"}
Sep 19 11:00:01.025390 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:01.025368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-wkdqz" event={"ID":"6dff9a57-bbb1-42fb-86fe-fd45d0ba32fb","Type":"ContainerStarted","Data":"1ffb460b7f3d506cc334d1958996b039ca0519d8f85a12bfa3457c709270e596"}
Sep 19 11:00:01.617802 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:01.617748    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:00:02.390329 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:02.390280    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:02.537372 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:02.537324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:03.049068 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:03.049031    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:03.149139 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:03.149099    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:03.282869 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:03.282833    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:04.045955 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:04.045913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-wkdqz" event={"ID":"6dff9a57-bbb1-42fb-86fe-fd45d0ba32fb","Type":"ContainerStarted","Data":"b7bddd24c21402a8eb1c79ddc35e0a5b85064ff976a2404c02f3e4ac19833317"}
Sep 19 11:00:04.047541 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:04.047508    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb" event={"ID":"4f085bb5-808c-42d6-81da-cfdf6c8aa4b2","Type":"ContainerStarted","Data":"a2158e6760763350a04efc9b5003b32a7dedc35bc5706c4633defef74bca3e8e"}
Sep 19 11:00:04.049129 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:04.049110    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw" event={"ID":"c52b427c-5c69-49fc-a9b9-884cba258ec2","Type":"ContainerStarted","Data":"e6337e91b01ac2789ad7bf200fe45453524557da2a13d7b6667597ef033ad38c"}
Sep 19 11:00:04.061499 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:04.061445    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-wkdqz" podStartSLOduration=0.963800093 podStartE2EDuration="4.061424823s" totalImagesPullingTime="3.09762473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:00:04.058337579 +0000 UTC m=+1448.717082740" watchObservedRunningTime="2026-09-19 11:00:04.061424823 +0000 UTC m=+1448.720169985"
Sep 19 11:00:04.071058 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:04.071002    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw" podStartSLOduration=0.962712226 podStartE2EDuration="4.070984809s" totalImagesPullingTime="3.108272583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:00:04.07053494 +0000 UTC m=+1448.729280156" watchObservedRunningTime="2026-09-19 11:00:04.070984809 +0000 UTC m=+1448.729729968"
Sep 19 11:00:04.084021 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:04.083961    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb" podStartSLOduration=1.042839163 podStartE2EDuration="4.083943905s" totalImagesPullingTime="3.041104742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:00:04.082737304 +0000 UTC m=+1448.741482475" watchObservedRunningTime="2026-09-19 11:00:04.083943905 +0000 UTC m=+1448.742689065"
Sep 19 11:00:05.055626 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:05.055589    2580 generic.go:356] "Generic (PLEG): container finished" podID="4f085bb5-808c-42d6-81da-cfdf6c8aa4b2" containerID="a2158e6760763350a04efc9b5003b32a7dedc35bc5706c4633defef74bca3e8e" exitCode=0
Sep 19 11:00:05.056082 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:05.055683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb" event={"ID":"4f085bb5-808c-42d6-81da-cfdf6c8aa4b2","Type":"ContainerDied","Data":"a2158e6760763350a04efc9b5003b32a7dedc35bc5706c4633defef74bca3e8e"}
Sep 19 11:00:05.057264 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:05.057238    2580 generic.go:356] "Generic (PLEG): container finished" podID="c52b427c-5c69-49fc-a9b9-884cba258ec2" containerID="e6337e91b01ac2789ad7bf200fe45453524557da2a13d7b6667597ef033ad38c" exitCode=0
Sep 19 11:00:05.057362 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:05.057315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw" event={"ID":"c52b427c-5c69-49fc-a9b9-884cba258ec2","Type":"ContainerDied","Data":"e6337e91b01ac2789ad7bf200fe45453524557da2a13d7b6667597ef033ad38c"}
Sep 19 11:00:05.059053 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:05.059028    2580 generic.go:356] "Generic (PLEG): container finished" podID="6dff9a57-bbb1-42fb-86fe-fd45d0ba32fb" containerID="b7bddd24c21402a8eb1c79ddc35e0a5b85064ff976a2404c02f3e4ac19833317" exitCode=0
Sep 19 11:00:05.059165 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:05.059128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-wkdqz" event={"ID":"6dff9a57-bbb1-42fb-86fe-fd45d0ba32fb","Type":"ContainerDied","Data":"b7bddd24c21402a8eb1c79ddc35e0a5b85064ff976a2404c02f3e4ac19833317"}
Sep 19 11:00:05.282041 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:05.282007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:05.832985 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:05.832949    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:05.933623 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:05.933592    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:06.034823 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:06.034749    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:06.239367 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:06.239343    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb"
Sep 19 11:00:06.242445 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:06.242425    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw"
Sep 19 11:00:06.293929 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:06.293902    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-wkdqz"
Sep 19 11:00:06.724314 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:06.724276    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:07.072298 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:07.072210    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-wkdqz"
Sep 19 11:00:07.072475 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:07.072207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-wkdqz" event={"ID":"6dff9a57-bbb1-42fb-86fe-fd45d0ba32fb","Type":"ContainerDied","Data":"1ffb460b7f3d506cc334d1958996b039ca0519d8f85a12bfa3457c709270e596"}
Sep 19 11:00:07.072475 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:07.072362    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ffb460b7f3d506cc334d1958996b039ca0519d8f85a12bfa3457c709270e596"
Sep 19 11:00:07.073802 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:07.073746    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb" event={"ID":"4f085bb5-808c-42d6-81da-cfdf6c8aa4b2","Type":"ContainerDied","Data":"297200a5b15201a7e22c83f55a3d2fb25e572f015b9c5b5c3aa870cc19e1744f"}
Sep 19 11:00:07.073938 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:07.073834    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="297200a5b15201a7e22c83f55a3d2fb25e572f015b9c5b5c3aa870cc19e1744f"
Sep 19 11:00:07.073938 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:07.073783    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb"
Sep 19 11:00:07.075392 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:07.075365    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw" event={"ID":"c52b427c-5c69-49fc-a9b9-884cba258ec2","Type":"ContainerDied","Data":"25c13095bf0d00c086f77abbd6029a9998d5c8c1de7670ae096e3b14c27d8973"}
Sep 19 11:00:07.075549 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:07.075415    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25c13095bf0d00c086f77abbd6029a9998d5c8c1de7670ae096e3b14c27d8973"
Sep 19 11:00:07.075549 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:07.075428    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw"
Sep 19 11:00:07.366216 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:07.366122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:08.222331 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:08.222291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:08.877294 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:08.877251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:09.486805 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:09.486748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:10.111577 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.111531    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"]
Sep 19 11:00:10.112074 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.111936    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f085bb5-808c-42d6-81da-cfdf6c8aa4b2" containerName="cleanup"
Sep 19 11:00:10.112074 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.111947    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f085bb5-808c-42d6-81da-cfdf6c8aa4b2" containerName="cleanup"
Sep 19 11:00:10.112074 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.111957    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52b427c-5c69-49fc-a9b9-884cba258ec2" containerName="cleanup"
Sep 19 11:00:10.112074 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.111963    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52b427c-5c69-49fc-a9b9-884cba258ec2" containerName="cleanup"
Sep 19 11:00:10.112074 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.111974    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dff9a57-bbb1-42fb-86fe-fd45d0ba32fb" containerName="cleanup"
Sep 19 11:00:10.112074 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.111982    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dff9a57-bbb1-42fb-86fe-fd45d0ba32fb" containerName="cleanup"
Sep 19 11:00:10.112074 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.112046    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52b427c-5c69-49fc-a9b9-884cba258ec2" containerName="cleanup"
Sep 19 11:00:10.112074 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.112054    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dff9a57-bbb1-42fb-86fe-fd45d0ba32fb" containerName="cleanup"
Sep 19 11:00:10.112074 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.112061    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f085bb5-808c-42d6-81da-cfdf6c8aa4b2" containerName="cleanup"
Sep 19 11:00:10.115337 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.115316    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.117343 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.117322    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-000aa3-serving-cert\""
Sep 19 11:00:10.125323 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.125298    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"]
Sep 19 11:00:10.139057 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.139019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:10.241296 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.241265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:10.261991 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.261954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47f9f3a0-34f8-46dc-898c-459f5dd7004c-maas-api-tls\") pod \"maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.262147 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.262012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f9f3a0-34f8-46dc-898c-459f5dd7004c-openshift-service-ca\") pod \"maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.262147 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.262042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47f9f3a0-34f8-46dc-898c-459f5dd7004c-metrics-certs\") pod \"maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.262147 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.262104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxqsc\" (UniqueName: \"kubernetes.io/projected/47f9f3a0-34f8-46dc-898c-459f5dd7004c-kube-api-access-qxqsc\") pod \"maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.363290 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.363188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47f9f3a0-34f8-46dc-898c-459f5dd7004c-maas-api-tls\") pod \"maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.363290 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.363255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f9f3a0-34f8-46dc-898c-459f5dd7004c-openshift-service-ca\") pod \"maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.363290 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.363283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47f9f3a0-34f8-46dc-898c-459f5dd7004c-metrics-certs\") pod \"maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.363576 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.363338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxqsc\" (UniqueName: \"kubernetes.io/projected/47f9f3a0-34f8-46dc-898c-459f5dd7004c-kube-api-access-qxqsc\") pod \"maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.364631 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.364602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f9f3a0-34f8-46dc-898c-459f5dd7004c-openshift-service-ca\") pod \"maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.375025 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.374991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47f9f3a0-34f8-46dc-898c-459f5dd7004c-maas-api-tls\") pod \"maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.375948 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.375923    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47f9f3a0-34f8-46dc-898c-459f5dd7004c-metrics-certs\") pod \"maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.376246 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.376216    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:10.376770 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.376728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxqsc\" (UniqueName: \"kubernetes.io/projected/47f9f3a0-34f8-46dc-898c-459f5dd7004c-kube-api-access-qxqsc\") pod \"maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.428648 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.428612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:10.513896 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.513858    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:10.615206 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.615111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:10.681616 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.681579    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:10.706363 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:10.706334    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"]
Sep 19 11:00:10.709334 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:00:10.709306    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f9f3a0_34f8_46dc_898c_459f5dd7004c.slice/crio-ae09cc43f8999cbf328f69a8c512d0698fcbdf9bfc122f1d3f9521255a5502c8 WatchSource:0}: Error finding container ae09cc43f8999cbf328f69a8c512d0698fcbdf9bfc122f1d3f9521255a5502c8: Status 404 returned error can't find the container with id ae09cc43f8999cbf328f69a8c512d0698fcbdf9bfc122f1d3f9521255a5502c8
Sep 19 11:00:11.098717 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:11.098684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2" event={"ID":"47f9f3a0-34f8-46dc-898c-459f5dd7004c","Type":"ContainerStarted","Data":"ae09cc43f8999cbf328f69a8c512d0698fcbdf9bfc122f1d3f9521255a5502c8"}
Sep 19 11:00:12.107171 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:12.107122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2" event={"ID":"47f9f3a0-34f8-46dc-898c-459f5dd7004c","Type":"ContainerStarted","Data":"24d8560cac7f2388d1d8e285974dcf49d6bfafe92c433fb20bde4a1e64313d81"}
Sep 19 11:00:12.107549 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:12.107198    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:12.124547 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:12.124495    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2" podStartSLOduration=1.759660217 podStartE2EDuration="2.12447854s" totalImagesPullingTime="364.818323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:00:12.122537394 +0000 UTC m=+1456.781282544" watchObservedRunningTime="2026-09-19 11:00:12.12447854 +0000 UTC m=+1456.783223746"
Sep 19 11:00:18.119024 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:18.118998    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:00:26.220775 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:26.220726    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:00:29.396962 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.396921    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"]
Sep 19 11:00:29.400743 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.400721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.402930 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.402906    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1907c1\"/\"embedding-w0-1907c1-kserve-self-signed-certs\""
Sep 19 11:00:29.410224 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.410200    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"]
Sep 19 11:00:29.451793 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.451718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-home\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.452003 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.451801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6bpv\" (UniqueName: \"kubernetes.io/projected/df00f0e9-db3d-4496-bbf3-5156402deab9-kube-api-access-g6bpv\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.452003 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.451859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-tmp-dir\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.452003 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.451928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-dshm\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.452003 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.451987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-model-cache\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.452196 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.452033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df00f0e9-db3d-4496-bbf3-5156402deab9-tls-certs\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.552724 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.552677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-model-cache\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.552724 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.552730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df00f0e9-db3d-4496-bbf3-5156402deab9-tls-certs\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.552983 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.552954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-home\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.553038 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.553008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6bpv\" (UniqueName: \"kubernetes.io/projected/df00f0e9-db3d-4496-bbf3-5156402deab9-kube-api-access-g6bpv\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.553096 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.553086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-tmp-dir\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.553158 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.553141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-dshm\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.553263 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.553238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-model-cache\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.553336 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.553291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-home\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.553525 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.553495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-tmp-dir\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.555511 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.555477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-dshm\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.555569 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.555531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df00f0e9-db3d-4496-bbf3-5156402deab9-tls-certs\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.561039 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.561007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6bpv\" (UniqueName: \"kubernetes.io/projected/df00f0e9-db3d-4496-bbf3-5156402deab9-kube-api-access-g6bpv\") pod \"embedding-w0-1907c1-kserve-75f7998d69-bwxxc\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") " pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.602730 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.602696    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:29.711376 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.711346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:29.984855 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:29.984825    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"]
Sep 19 11:00:29.986564 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:00:29.986523    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf00f0e9_db3d_4496_bbf3_5156402deab9.slice/crio-f11b4627ca9d6ff4620c3291294e5b16743d66bb9d6f6870d81474a8bb56b706 WatchSource:0}: Error finding container f11b4627ca9d6ff4620c3291294e5b16743d66bb9d6f6870d81474a8bb56b706: Status 404 returned error can't find the container with id f11b4627ca9d6ff4620c3291294e5b16743d66bb9d6f6870d81474a8bb56b706
Sep 19 11:00:30.206227 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:30.206169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc" event={"ID":"df00f0e9-db3d-4496-bbf3-5156402deab9","Type":"ContainerStarted","Data":"adcbef3de067badf930edbd5151e117726f1874e3c5b4abc648515c96d138622"}
Sep 19 11:00:30.206227 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:30.206224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc" event={"ID":"df00f0e9-db3d-4496-bbf3-5156402deab9","Type":"ContainerStarted","Data":"f11b4627ca9d6ff4620c3291294e5b16743d66bb9d6f6870d81474a8bb56b706"}
Sep 19 11:00:30.222008 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:30.221935    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc" podStartSLOduration=1.221914801 podStartE2EDuration="1.221914801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:00:30.220607125 +0000 UTC m=+1474.879352285" watchObservedRunningTime="2026-09-19 11:00:30.221914801 +0000 UTC m=+1474.880659961"
Sep 19 11:00:32.328823 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:32.328788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:33.094989 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:33.094949    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:33.714076 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:33.714041    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:34.333489 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:34.333453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:35.101125 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:35.101089    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:35.911009 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:35.910976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:36.011716 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:36.011679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:36.519990 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:36.519952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:36.750213 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:36.750174    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:36.814880 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:36.814745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:37.160064 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:37.159973    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:37.733586 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:37.733543    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:38.056990 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:38.056907    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:39.415059 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:39.415021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:39.711964 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:39.711927    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:39.712151 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:39.711991    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:39.724838 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:39.724803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:40.080980 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:40.080885    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:40.173037 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:40.173004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:40.272974 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:40.272937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:40.275939 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:40.275917    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:00:41.265394 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:41.265360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:42.033276 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:42.033236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:42.804875 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:42.804839    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:43.430057 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:43.430017    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:44.111777 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:44.111723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:44.824924 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:44.824891    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:45.551502 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:45.551460    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:45.652210 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:45.652173    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:45.699065 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:45.699023    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:46.019773 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.019718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:46.124033 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.123999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:00:46.160857 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.160819    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"]
Sep 19 11:00:46.164725 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.164701    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"
Sep 19 11:00:46.166959 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.166929    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0c8f26-dockercfg-rb2wg\""
Sep 19 11:00:46.167090 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.166981    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0c8f26\""
Sep 19 11:00:46.173527 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.173206    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"]
Sep 19 11:00:46.219811 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.219779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ccf\" (UniqueName: \"kubernetes.io/projected/63b01b51-bde4-46c3-b747-75f037aa78b5-kube-api-access-m7ccf\") pod \"payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87\" (UID: \"63b01b51-bde4-46c3-b747-75f037aa78b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"
Sep 19 11:00:46.219811 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.219823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b01b51-bde4-46c3-b747-75f037aa78b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87\" (UID: \"63b01b51-bde4-46c3-b747-75f037aa78b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"
Sep 19 11:00:46.321253 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.321153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b01b51-bde4-46c3-b747-75f037aa78b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87\" (UID: \"63b01b51-bde4-46c3-b747-75f037aa78b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"
Sep 19 11:00:46.321446 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.321295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ccf\" (UniqueName: \"kubernetes.io/projected/63b01b51-bde4-46c3-b747-75f037aa78b5-kube-api-access-m7ccf\") pod \"payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87\" (UID: \"63b01b51-bde4-46c3-b747-75f037aa78b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"
Sep 19 11:00:46.321954 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.321929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b01b51-bde4-46c3-b747-75f037aa78b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87\" (UID: \"63b01b51-bde4-46c3-b747-75f037aa78b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"
Sep 19 11:00:46.329075 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.329046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ccf\" (UniqueName: \"kubernetes.io/projected/63b01b51-bde4-46c3-b747-75f037aa78b5-kube-api-access-m7ccf\") pod \"payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87\" (UID: \"63b01b51-bde4-46c3-b747-75f037aa78b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"
Sep 19 11:00:46.477675 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.477635    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"
Sep 19 11:00:46.755403 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:46.755374    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"]
Sep 19 11:00:46.757555 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:00:46.757513    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b01b51_bde4_46c3_b747_75f037aa78b5.slice/crio-0a41ceedfb13d72806765a454acdc17e1565fc5825367740b5d6345632643d82 WatchSource:0}: Error finding container 0a41ceedfb13d72806765a454acdc17e1565fc5825367740b5d6345632643d82: Status 404 returned error can't find the container with id 0a41ceedfb13d72806765a454acdc17e1565fc5825367740b5d6345632643d82
Sep 19 11:00:47.305528 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:47.305495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87" event={"ID":"63b01b51-bde4-46c3-b747-75f037aa78b5","Type":"ContainerStarted","Data":"4bfbc162ec5d3ae0c9bad7e11fd732d5a90f92491626f249e5696858b4cab5ad"}
Sep 19 11:00:47.305726 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:47.305545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87" event={"ID":"63b01b51-bde4-46c3-b747-75f037aa78b5","Type":"ContainerStarted","Data":"0a41ceedfb13d72806765a454acdc17e1565fc5825367740b5d6345632643d82"}
Sep 19 11:00:47.305726 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:47.305646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"
Sep 19 11:00:47.320632 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:47.320567    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87" podStartSLOduration=1.320542721 podStartE2EDuration="1.320542721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:00:47.319800719 +0000 UTC m=+1491.978545879" watchObservedRunningTime="2026-09-19 11:00:47.320542721 +0000 UTC m=+1491.979287886"
Sep 19 11:00:55.906877 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:55.906848    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 11:00:55.907375 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:55.906895    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 11:00:58.313124 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:00:58.313095    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"
Sep 19 11:01:07.418910 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:07.418875    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:08.127139 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:08.127101    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:08.861447 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:08.861408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:09.022814 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:09.022780    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:01:09.694600 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:09.694563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:10.165290 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:10.165188    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:10.416437 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:10.416354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:11.316376 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:11.316339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:12.108418 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:12.108382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:12.923689 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:12.923654    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:13.734335 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:13.734294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:14.437849 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:14.437816    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:15.346098 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:15.346061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:16.083300 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:16.083265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:16.719646 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:16.719562    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:17.528039 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:17.528002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:18.148534 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:18.148483    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:19.010980 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:19.010949    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:19.725267 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:19.725215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:23.769772 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:23.769711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:23.870444 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:23.870400    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:27.498600 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:27.498566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:28.744329 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:28.744293    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"]
Sep 19 11:01:28.744874 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:28.744509    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2" podUID="47f9f3a0-34f8-46dc-898c-459f5dd7004c" containerName="maas-api" containerID="cri-o://24d8560cac7f2388d1d8e285974dcf49d6bfafe92c433fb20bde4a1e64313d81" gracePeriod=30
Sep 19 11:01:28.775543 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:28.775481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:28.843688 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:28.843641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:28.913302 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:28.913265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:28.971692 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:28.971646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:29.000690 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.000623    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:01:29.136794 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.136737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxqsc\" (UniqueName: \"kubernetes.io/projected/47f9f3a0-34f8-46dc-898c-459f5dd7004c-kube-api-access-qxqsc\") pod \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") "
Sep 19 11:01:29.137007 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.136896    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47f9f3a0-34f8-46dc-898c-459f5dd7004c-maas-api-tls\") pod \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") "
Sep 19 11:01:29.137007 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.136952    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f9f3a0-34f8-46dc-898c-459f5dd7004c-openshift-service-ca\") pod \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") "
Sep 19 11:01:29.137007 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.137003    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47f9f3a0-34f8-46dc-898c-459f5dd7004c-metrics-certs\") pod \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\" (UID: \"47f9f3a0-34f8-46dc-898c-459f5dd7004c\") "
Sep 19 11:01:29.137410 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.137358    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f9f3a0-34f8-46dc-898c-459f5dd7004c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47f9f3a0-34f8-46dc-898c-459f5dd7004c" (UID: "47f9f3a0-34f8-46dc-898c-459f5dd7004c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:01:29.139950 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.139914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f9f3a0-34f8-46dc-898c-459f5dd7004c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47f9f3a0-34f8-46dc-898c-459f5dd7004c" (UID: "47f9f3a0-34f8-46dc-898c-459f5dd7004c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:01:29.139950 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.139931    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f9f3a0-34f8-46dc-898c-459f5dd7004c-kube-api-access-qxqsc" (OuterVolumeSpecName: "kube-api-access-qxqsc") pod "47f9f3a0-34f8-46dc-898c-459f5dd7004c" (UID: "47f9f3a0-34f8-46dc-898c-459f5dd7004c"). InnerVolumeSpecName "kube-api-access-qxqsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:01:29.139950 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.139932    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f9f3a0-34f8-46dc-898c-459f5dd7004c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "47f9f3a0-34f8-46dc-898c-459f5dd7004c" (UID: "47f9f3a0-34f8-46dc-898c-459f5dd7004c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:01:29.238732 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.238699    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47f9f3a0-34f8-46dc-898c-459f5dd7004c-maas-api-tls\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:01:29.238732 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.238729    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f9f3a0-34f8-46dc-898c-459f5dd7004c-openshift-service-ca\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:01:29.238732 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.238738    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47f9f3a0-34f8-46dc-898c-459f5dd7004c-metrics-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:01:29.238980 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.238746    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxqsc\" (UniqueName: \"kubernetes.io/projected/47f9f3a0-34f8-46dc-898c-459f5dd7004c-kube-api-access-qxqsc\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:01:29.314180 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.314083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:29.440519 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.440479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:29.531903 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.531874    2580 generic.go:356] "Generic (PLEG): container finished" podID="47f9f3a0-34f8-46dc-898c-459f5dd7004c" containerID="24d8560cac7f2388d1d8e285974dcf49d6bfafe92c433fb20bde4a1e64313d81" exitCode=0
Sep 19 11:01:29.532081 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.531917    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"
Sep 19 11:01:29.532081 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.531968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2" event={"ID":"47f9f3a0-34f8-46dc-898c-459f5dd7004c","Type":"ContainerDied","Data":"24d8560cac7f2388d1d8e285974dcf49d6bfafe92c433fb20bde4a1e64313d81"}
Sep 19 11:01:29.532081 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.532026    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2" event={"ID":"47f9f3a0-34f8-46dc-898c-459f5dd7004c","Type":"ContainerDied","Data":"ae09cc43f8999cbf328f69a8c512d0698fcbdf9bfc122f1d3f9521255a5502c8"}
Sep 19 11:01:29.532195 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.532054    2580 scope.go:117] "RemoveContainer" containerID="24d8560cac7f2388d1d8e285974dcf49d6bfafe92c433fb20bde4a1e64313d81"
Sep 19 11:01:29.542309 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.542288    2580 scope.go:117] "RemoveContainer" containerID="24d8560cac7f2388d1d8e285974dcf49d6bfafe92c433fb20bde4a1e64313d81"
Sep 19 11:01:29.542609 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:01:29.542588    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d8560cac7f2388d1d8e285974dcf49d6bfafe92c433fb20bde4a1e64313d81\": container with ID starting with 24d8560cac7f2388d1d8e285974dcf49d6bfafe92c433fb20bde4a1e64313d81 not found: ID does not exist" containerID="24d8560cac7f2388d1d8e285974dcf49d6bfafe92c433fb20bde4a1e64313d81"
Sep 19 11:01:29.542690 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.542619    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d8560cac7f2388d1d8e285974dcf49d6bfafe92c433fb20bde4a1e64313d81"} err="failed to get container status \"24d8560cac7f2388d1d8e285974dcf49d6bfafe92c433fb20bde4a1e64313d81\": rpc error: code = NotFound desc = could not find container \"24d8560cac7f2388d1d8e285974dcf49d6bfafe92c433fb20bde4a1e64313d81\": container with ID starting with 24d8560cac7f2388d1d8e285974dcf49d6bfafe92c433fb20bde4a1e64313d81 not found: ID does not exist"
Sep 19 11:01:29.552382 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.552359    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"]
Sep 19 11:01:29.556203 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.556183    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-000aa3-9cb9df7f8-gvzg2"]
Sep 19 11:01:29.857066 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:29.857033    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47f9f3a0-34f8-46dc-898c-459f5dd7004c" path="/var/lib/kubelet/pods/47f9f3a0-34f8-46dc-898c-459f5dd7004c/volumes"
Sep 19 11:01:30.073819 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:30.073788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:30.896365 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:30.896318    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:31.515335 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:31.515296    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:32.232343 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:32.232309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:33.007535 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:33.007500    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:33.621499 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:33.621412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:34.390505 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:34.390473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:35.284691 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:35.284653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:35.336434 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:35.336396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:35.537541 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:35.537443    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:36.031100 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:36.031064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:36.815935 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:36.815893    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:37.527561 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:37.527524    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:37.577628 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:37.577593    2580 generic.go:356] "Generic (PLEG): container finished" podID="727bbfa8-e3c8-4f40-b05f-f31161f3b1ad" containerID="3ba3c1413698a10426615ca08fdfd54a10e05d268761f6dc895f30471734780a" exitCode=137
Sep 19 11:01:37.577830 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:37.577644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z" event={"ID":"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad","Type":"ContainerDied","Data":"3ba3c1413698a10426615ca08fdfd54a10e05d268761f6dc895f30471734780a"}
Sep 19 11:01:37.578062 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:37.578041    2580 scope.go:117] "RemoveContainer" containerID="3ba3c1413698a10426615ca08fdfd54a10e05d268761f6dc895f30471734780a"
Sep 19 11:01:38.001038 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:38.000996    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 11:01:38.239838 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:38.239802    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:38.585504 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:38.585469    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z" event={"ID":"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad","Type":"ContainerStarted","Data":"9f9e90a6f41a8d0acc343900e7db93b8a94b182bd4e6144118babc227e870610"}
Sep 19 11:01:38.585700 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:38.585593    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 11:01:39.025361 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:39.025325    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:39.923227 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:39.923191    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:40.609117 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:40.609078    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:41.315130 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:41.315101    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:42.024364 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:42.024317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:45.838522 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:45.838480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:49.593774 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:49.593732    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 11:01:51.410529 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.410486    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:51.486883 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.486844    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"]
Sep 19 11:01:51.487111 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.487079    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87" podUID="63b01b51-bde4-46c3-b747-75f037aa78b5" containerName="payload-processing" containerID="cri-o://4bfbc162ec5d3ae0c9bad7e11fd732d5a90f92491626f249e5696858b4cab5ad" gracePeriod=30
Sep 19 11:01:51.532866 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.532824    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:51.635695 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.635663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:51.657818 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.657782    2580 generic.go:356] "Generic (PLEG): container finished" podID="63b01b51-bde4-46c3-b747-75f037aa78b5" containerID="4bfbc162ec5d3ae0c9bad7e11fd732d5a90f92491626f249e5696858b4cab5ad" exitCode=0
Sep 19 11:01:51.658114 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.657973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87" event={"ID":"63b01b51-bde4-46c3-b747-75f037aa78b5","Type":"ContainerDied","Data":"4bfbc162ec5d3ae0c9bad7e11fd732d5a90f92491626f249e5696858b4cab5ad"}
Sep 19 11:01:51.733801 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.733770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:51.751492 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.751468    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"
Sep 19 11:01:51.850306 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.850268    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b01b51-bde4-46c3-b747-75f037aa78b5-plugins-config-volume\") pod \"63b01b51-bde4-46c3-b747-75f037aa78b5\" (UID: \"63b01b51-bde4-46c3-b747-75f037aa78b5\") "
Sep 19 11:01:51.850523 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.850369    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7ccf\" (UniqueName: \"kubernetes.io/projected/63b01b51-bde4-46c3-b747-75f037aa78b5-kube-api-access-m7ccf\") pod \"63b01b51-bde4-46c3-b747-75f037aa78b5\" (UID: \"63b01b51-bde4-46c3-b747-75f037aa78b5\") "
Sep 19 11:01:51.850724 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.850700    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b01b51-bde4-46c3-b747-75f037aa78b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63b01b51-bde4-46c3-b747-75f037aa78b5" (UID: "63b01b51-bde4-46c3-b747-75f037aa78b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:01:51.852697 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.852674    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b01b51-bde4-46c3-b747-75f037aa78b5-kube-api-access-m7ccf" (OuterVolumeSpecName: "kube-api-access-m7ccf") pod "63b01b51-bde4-46c3-b747-75f037aa78b5" (UID: "63b01b51-bde4-46c3-b747-75f037aa78b5"). InnerVolumeSpecName "kube-api-access-m7ccf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:01:51.951703 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.951669    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b01b51-bde4-46c3-b747-75f037aa78b5-plugins-config-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:01:51.951703 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:51.951701    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7ccf\" (UniqueName: \"kubernetes.io/projected/63b01b51-bde4-46c3-b747-75f037aa78b5-kube-api-access-m7ccf\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:01:52.002810 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:52.002676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:52.130464 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:52.130434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:52.665137 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:52.665104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87" event={"ID":"63b01b51-bde4-46c3-b747-75f037aa78b5","Type":"ContainerDied","Data":"0a41ceedfb13d72806765a454acdc17e1565fc5825367740b5d6345632643d82"}
Sep 19 11:01:52.665137 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:52.665133    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"
Sep 19 11:01:52.665638 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:52.665163    2580 scope.go:117] "RemoveContainer" containerID="4bfbc162ec5d3ae0c9bad7e11fd732d5a90f92491626f249e5696858b4cab5ad"
Sep 19 11:01:52.681554 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:52.681519    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"]
Sep 19 11:01:52.685340 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:52.685313    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0c8f26-58b8c67447-xqh87"]
Sep 19 11:01:52.805615 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:52.805578    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:53.528659 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:53.528614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:53.857093 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:53.857010    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63b01b51-bde4-46c3-b747-75f037aa78b5" path="/var/lib/kubelet/pods/63b01b51-bde4-46c3-b747-75f037aa78b5/volumes"
Sep 19 11:01:54.254445 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:54.254411    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:54.870634 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:54.870599    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:55.722634 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:55.722596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:56.356364 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:56.356270    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:56.979392 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:56.979357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:57.856916 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:57.856875    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:57.957818 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:57.957745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:58.109988 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:58.109898    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:58.713692 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:58.713653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:01:59.429141 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:01:59.429101    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:00.104667 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:00.104631    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:00.830287 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:00.830254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:01.518187 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:01.518140    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:02.144721 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:02.144680    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:03.011067 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:03.011036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:03.636880 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:03.636842    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:08.727861 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:08.727825    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:02:08.935948 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:08.935909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:09.000893 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:09.000789    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:09.058739 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:09.058687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:09.125399 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:09.125357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:09.187367 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:09.187332    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:09.490538 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:09.490505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:09.591186 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:09.591148    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:10.236588 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:10.236551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:10.926397 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:10.926362    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:11.574288 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:11.574242    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:12.298858 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:12.298806    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:13.103997 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:13.103956    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:15.312218 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:15.312167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:15.369422 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:15.369381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:15.552802 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:15.552745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:16.032876 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:16.032840    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:16.788012 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:16.787975    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:17.509733 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:17.509693    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:18.123213 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:18.123171    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:18.738035 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:18.738001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:19.513032 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:19.512997    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:20.124622 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:20.124590    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:26.327305 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.327269    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw"]
Sep 19 11:02:26.329422 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.329398    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8ddce5-29830260-96ktw"]
Sep 19 11:02:26.329565 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.329503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:26.355548 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.355511    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"]
Sep 19 11:02:26.355809 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.355779    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h" podUID="195d93e0-1853-490e-8d78-e48fc35a4021" containerName="payload-processing" containerID="cri-o://4da5f86c318e37c11742f7aecea31795bf83c7ba43bf51aba1e14511ba2fe02d" gracePeriod=30
Sep 19 11:02:26.362829 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.360165    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"]
Sep 19 11:02:26.363322 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.363272    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r" podUID="971fea77-05d6-4ce7-b005-85f93a5f727d" containerName="payload-pre-processing" containerID="cri-o://ae990377c792c2b4dc17556ea9c7ca84957390255d831713eec94faa26b5dce3" gracePeriod=30
Sep 19 11:02:26.396151 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.396118    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:26.525386 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.525347    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:26.614771 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.614715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:26.625301 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.625274    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"
Sep 19 11:02:26.645610 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.645581    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"
Sep 19 11:02:26.704143 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.704104    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195d93e0-1853-490e-8d78-e48fc35a4021-plugins-config-volume\") pod \"195d93e0-1853-490e-8d78-e48fc35a4021\" (UID: \"195d93e0-1853-490e-8d78-e48fc35a4021\") "
Sep 19 11:02:26.704337 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.704158    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971fea77-05d6-4ce7-b005-85f93a5f727d-plugins-config-volume\") pod \"971fea77-05d6-4ce7-b005-85f93a5f727d\" (UID: \"971fea77-05d6-4ce7-b005-85f93a5f727d\") "
Sep 19 11:02:26.704337 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.704182    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k44q4\" (UniqueName: \"kubernetes.io/projected/971fea77-05d6-4ce7-b005-85f93a5f727d-kube-api-access-k44q4\") pod \"971fea77-05d6-4ce7-b005-85f93a5f727d\" (UID: \"971fea77-05d6-4ce7-b005-85f93a5f727d\") "
Sep 19 11:02:26.704337 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.704255    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwtlp\" (UniqueName: \"kubernetes.io/projected/195d93e0-1853-490e-8d78-e48fc35a4021-kube-api-access-pwtlp\") pod \"195d93e0-1853-490e-8d78-e48fc35a4021\" (UID: \"195d93e0-1853-490e-8d78-e48fc35a4021\") "
Sep 19 11:02:26.704592 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.704566    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971fea77-05d6-4ce7-b005-85f93a5f727d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "971fea77-05d6-4ce7-b005-85f93a5f727d" (UID: "971fea77-05d6-4ce7-b005-85f93a5f727d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:02:26.704652 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.704567    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195d93e0-1853-490e-8d78-e48fc35a4021-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "195d93e0-1853-490e-8d78-e48fc35a4021" (UID: "195d93e0-1853-490e-8d78-e48fc35a4021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:02:26.706396 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.706371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971fea77-05d6-4ce7-b005-85f93a5f727d-kube-api-access-k44q4" (OuterVolumeSpecName: "kube-api-access-k44q4") pod "971fea77-05d6-4ce7-b005-85f93a5f727d" (UID: "971fea77-05d6-4ce7-b005-85f93a5f727d"). InnerVolumeSpecName "kube-api-access-k44q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:02:26.706483 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.706411    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195d93e0-1853-490e-8d78-e48fc35a4021-kube-api-access-pwtlp" (OuterVolumeSpecName: "kube-api-access-pwtlp") pod "195d93e0-1853-490e-8d78-e48fc35a4021" (UID: "195d93e0-1853-490e-8d78-e48fc35a4021"). InnerVolumeSpecName "kube-api-access-pwtlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:02:26.811202 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.805781    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwtlp\" (UniqueName: \"kubernetes.io/projected/195d93e0-1853-490e-8d78-e48fc35a4021-kube-api-access-pwtlp\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:26.811202 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.805838    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195d93e0-1853-490e-8d78-e48fc35a4021-plugins-config-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:26.811202 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.805855    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/971fea77-05d6-4ce7-b005-85f93a5f727d-plugins-config-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:26.811202 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.805868    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k44q4\" (UniqueName: \"kubernetes.io/projected/971fea77-05d6-4ce7-b005-85f93a5f727d-kube-api-access-k44q4\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:26.849343 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.849241    2580 generic.go:356] "Generic (PLEG): container finished" podID="195d93e0-1853-490e-8d78-e48fc35a4021" containerID="4da5f86c318e37c11742f7aecea31795bf83c7ba43bf51aba1e14511ba2fe02d" exitCode=0
Sep 19 11:02:26.849343 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.849295    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"
Sep 19 11:02:26.849343 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.849322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h" event={"ID":"195d93e0-1853-490e-8d78-e48fc35a4021","Type":"ContainerDied","Data":"4da5f86c318e37c11742f7aecea31795bf83c7ba43bf51aba1e14511ba2fe02d"}
Sep 19 11:02:26.849609 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.849406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h" event={"ID":"195d93e0-1853-490e-8d78-e48fc35a4021","Type":"ContainerDied","Data":"dc04d167473612b658f722416dd9678d4cc3e8b21663711e5177dbcace7e77e1"}
Sep 19 11:02:26.849609 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.849467    2580 scope.go:117] "RemoveContainer" containerID="4da5f86c318e37c11742f7aecea31795bf83c7ba43bf51aba1e14511ba2fe02d"
Sep 19 11:02:26.850802 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.850778    2580 generic.go:356] "Generic (PLEG): container finished" podID="971fea77-05d6-4ce7-b005-85f93a5f727d" containerID="ae990377c792c2b4dc17556ea9c7ca84957390255d831713eec94faa26b5dce3" exitCode=0
Sep 19 11:02:26.850894 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.850820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r" event={"ID":"971fea77-05d6-4ce7-b005-85f93a5f727d","Type":"ContainerDied","Data":"ae990377c792c2b4dc17556ea9c7ca84957390255d831713eec94faa26b5dce3"}
Sep 19 11:02:26.850894 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.850875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r" event={"ID":"971fea77-05d6-4ce7-b005-85f93a5f727d","Type":"ContainerDied","Data":"7a7439ba60e79e25a24cab2757ea20e744809d0d1102570c5d61ad101c1084b6"}
Sep 19 11:02:26.851005 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.850832    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"
Sep 19 11:02:26.859404 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.859382    2580 scope.go:117] "RemoveContainer" containerID="4da5f86c318e37c11742f7aecea31795bf83c7ba43bf51aba1e14511ba2fe02d"
Sep 19 11:02:26.859843 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:02:26.859819    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da5f86c318e37c11742f7aecea31795bf83c7ba43bf51aba1e14511ba2fe02d\": container with ID starting with 4da5f86c318e37c11742f7aecea31795bf83c7ba43bf51aba1e14511ba2fe02d not found: ID does not exist" containerID="4da5f86c318e37c11742f7aecea31795bf83c7ba43bf51aba1e14511ba2fe02d"
Sep 19 11:02:26.859946 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.859854    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da5f86c318e37c11742f7aecea31795bf83c7ba43bf51aba1e14511ba2fe02d"} err="failed to get container status \"4da5f86c318e37c11742f7aecea31795bf83c7ba43bf51aba1e14511ba2fe02d\": rpc error: code = NotFound desc = could not find container \"4da5f86c318e37c11742f7aecea31795bf83c7ba43bf51aba1e14511ba2fe02d\": container with ID starting with 4da5f86c318e37c11742f7aecea31795bf83c7ba43bf51aba1e14511ba2fe02d not found: ID does not exist"
Sep 19 11:02:26.859946 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.859878    2580 scope.go:117] "RemoveContainer" containerID="ae990377c792c2b4dc17556ea9c7ca84957390255d831713eec94faa26b5dce3"
Sep 19 11:02:26.870764 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.870726    2580 scope.go:117] "RemoveContainer" containerID="ae990377c792c2b4dc17556ea9c7ca84957390255d831713eec94faa26b5dce3"
Sep 19 11:02:26.871093 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:02:26.871072    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae990377c792c2b4dc17556ea9c7ca84957390255d831713eec94faa26b5dce3\": container with ID starting with ae990377c792c2b4dc17556ea9c7ca84957390255d831713eec94faa26b5dce3 not found: ID does not exist" containerID="ae990377c792c2b4dc17556ea9c7ca84957390255d831713eec94faa26b5dce3"
Sep 19 11:02:26.871169 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.871098    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae990377c792c2b4dc17556ea9c7ca84957390255d831713eec94faa26b5dce3"} err="failed to get container status \"ae990377c792c2b4dc17556ea9c7ca84957390255d831713eec94faa26b5dce3\": rpc error: code = NotFound desc = could not find container \"ae990377c792c2b4dc17556ea9c7ca84957390255d831713eec94faa26b5dce3\": container with ID starting with ae990377c792c2b4dc17556ea9c7ca84957390255d831713eec94faa26b5dce3 not found: ID does not exist"
Sep 19 11:02:26.874823 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.874799    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"]
Sep 19 11:02:26.876594 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.876572    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8ddce5-575d9cbd9c-fp86h"]
Sep 19 11:02:26.885162 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.885135    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"]
Sep 19 11:02:26.889037 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.889010    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ddce5-6cc575ffc8-nsv2r"]
Sep 19 11:02:26.935338 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:26.935300    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:27.034011 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:27.033956    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:27.718438 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:27.718388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:27.856508 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:27.856469    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195d93e0-1853-490e-8d78-e48fc35a4021" path="/var/lib/kubelet/pods/195d93e0-1853-490e-8d78-e48fc35a4021/volumes"
Sep 19 11:02:27.857002 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:27.856984    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="971fea77-05d6-4ce7-b005-85f93a5f727d" path="/var/lib/kubelet/pods/971fea77-05d6-4ce7-b005-85f93a5f727d/volumes"
Sep 19 11:02:27.857328 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:27.857307    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c52b427c-5c69-49fc-a9b9-884cba258ec2" path="/var/lib/kubelet/pods/c52b427c-5c69-49fc-a9b9-884cba258ec2/volumes"
Sep 19 11:02:28.335493 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:28.335453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:29.013017 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:29.012978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:29.707907 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:29.707876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:32.826987 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:32.826942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:32.887491 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:32.887456    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:33.069197 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:33.069161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:34.724031 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:34.723990    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:38.954519 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:38.954475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:39.519515 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:39.519475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:40.308913 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:40.308876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:40.926292 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:40.926257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:41.590414 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:41.590379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:42.341012 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:42.340972    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:42.515036 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:42.514999    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:02:43.058318 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:43.058279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:43.940514 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:43.940479    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"]
Sep 19 11:02:43.956721 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:43.956691    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"]
Sep 19 11:02:43.965659 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:43.965632    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"]
Sep 19 11:02:43.975990 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:43.975963    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"]
Sep 19 11:02:44.020200 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:44.020162    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:48.888506 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:48.888420    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:49.194253 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:49.194216    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:50.130974 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:50.130941    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:50.824862 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:50.824821    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:51.507458 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:51.507420    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:52.211575 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:52.211540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:53.877477 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:53.877443    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"]
Sep 19 11:02:53.896029 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:53.895995    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"]
Sep 19 11:02:53.944646 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:53.944617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:55.019335 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.019300    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"]
Sep 19 11:02:55.019972 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.019608    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98" podUID="89353aaf-f742-434b-9b75-f6fde2b8babc" containerName="maas-api" containerID="cri-o://5d2035d6261ed7a391edeedcae11a0576c9031a104285a886f887f504aa75b20" gracePeriod=30
Sep 19 11:02:55.044476 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.044448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:55.108164 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.108131    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:55.205684 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.205647    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:55.272596 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.272531    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 11:02:55.292306 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.292262    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:55.379366 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.379334    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr4x9\" (UniqueName: \"kubernetes.io/projected/89353aaf-f742-434b-9b75-f6fde2b8babc-kube-api-access-tr4x9\") pod \"89353aaf-f742-434b-9b75-f6fde2b8babc\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") "
Sep 19 11:02:55.379578 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.379411    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-metrics-certs\") pod \"89353aaf-f742-434b-9b75-f6fde2b8babc\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") "
Sep 19 11:02:55.379578 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.379448    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89353aaf-f742-434b-9b75-f6fde2b8babc-openshift-service-ca\") pod \"89353aaf-f742-434b-9b75-f6fde2b8babc\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") "
Sep 19 11:02:55.379578 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.379531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-maas-api-tls\") pod \"89353aaf-f742-434b-9b75-f6fde2b8babc\" (UID: \"89353aaf-f742-434b-9b75-f6fde2b8babc\") "
Sep 19 11:02:55.379866 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.379831    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89353aaf-f742-434b-9b75-f6fde2b8babc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89353aaf-f742-434b-9b75-f6fde2b8babc" (UID: "89353aaf-f742-434b-9b75-f6fde2b8babc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:02:55.381735 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.381714    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "89353aaf-f742-434b-9b75-f6fde2b8babc" (UID: "89353aaf-f742-434b-9b75-f6fde2b8babc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:02:55.381827 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.381746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89353aaf-f742-434b-9b75-f6fde2b8babc" (UID: "89353aaf-f742-434b-9b75-f6fde2b8babc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:02:55.381915 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.381899    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89353aaf-f742-434b-9b75-f6fde2b8babc-kube-api-access-tr4x9" (OuterVolumeSpecName: "kube-api-access-tr4x9") pod "89353aaf-f742-434b-9b75-f6fde2b8babc" (UID: "89353aaf-f742-434b-9b75-f6fde2b8babc"). InnerVolumeSpecName "kube-api-access-tr4x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:02:55.481290 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.481250    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr4x9\" (UniqueName: \"kubernetes.io/projected/89353aaf-f742-434b-9b75-f6fde2b8babc-kube-api-access-tr4x9\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:55.481290 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.481282    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-metrics-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:55.481290 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.481292    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89353aaf-f742-434b-9b75-f6fde2b8babc-openshift-service-ca\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:55.481290 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.481300    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89353aaf-f742-434b-9b75-f6fde2b8babc-maas-api-tls\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:55.562306 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.562217    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:55.699920 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:55.699884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:56.024735 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:56.024701    2580 generic.go:356] "Generic (PLEG): container finished" podID="89353aaf-f742-434b-9b75-f6fde2b8babc" containerID="5d2035d6261ed7a391edeedcae11a0576c9031a104285a886f887f504aa75b20" exitCode=0
Sep 19 11:02:56.025162 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:56.024741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98" event={"ID":"89353aaf-f742-434b-9b75-f6fde2b8babc","Type":"ContainerDied","Data":"5d2035d6261ed7a391edeedcae11a0576c9031a104285a886f887f504aa75b20"}
Sep 19 11:02:56.025162 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:56.024778    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"
Sep 19 11:02:56.025162 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:56.024810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98" event={"ID":"89353aaf-f742-434b-9b75-f6fde2b8babc","Type":"ContainerDied","Data":"395e466b9b1b07537c67d61ab9c25d6e01bcffcc55169d1c4c111eb52f5ba1e3"}
Sep 19 11:02:56.025162 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:56.024832    2580 scope.go:117] "RemoveContainer" containerID="5d2035d6261ed7a391edeedcae11a0576c9031a104285a886f887f504aa75b20"
Sep 19 11:02:56.033907 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:56.033886    2580 scope.go:117] "RemoveContainer" containerID="5d2035d6261ed7a391edeedcae11a0576c9031a104285a886f887f504aa75b20"
Sep 19 11:02:56.034222 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:02:56.034205    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2035d6261ed7a391edeedcae11a0576c9031a104285a886f887f504aa75b20\": container with ID starting with 5d2035d6261ed7a391edeedcae11a0576c9031a104285a886f887f504aa75b20 not found: ID does not exist" containerID="5d2035d6261ed7a391edeedcae11a0576c9031a104285a886f887f504aa75b20"
Sep 19 11:02:56.034284 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:56.034233    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2035d6261ed7a391edeedcae11a0576c9031a104285a886f887f504aa75b20"} err="failed to get container status \"5d2035d6261ed7a391edeedcae11a0576c9031a104285a886f887f504aa75b20\": rpc error: code = NotFound desc = could not find container \"5d2035d6261ed7a391edeedcae11a0576c9031a104285a886f887f504aa75b20\": container with ID starting with 5d2035d6261ed7a391edeedcae11a0576c9031a104285a886f887f504aa75b20 not found: ID does not exist"
Sep 19 11:02:56.039488 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:56.039465    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"]
Sep 19 11:02:56.042796 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:56.042776    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1907c1-55dc6b6f46-cxh98"]
Sep 19 11:02:56.342212 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:56.342174    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:57.045428 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:57.045392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:57.689504 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:57.689470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:57.856528 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:57.856489    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89353aaf-f742-434b-9b75-f6fde2b8babc" path="/var/lib/kubelet/pods/89353aaf-f742-434b-9b75-f6fde2b8babc/volumes"
Sep 19 11:02:58.961567 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:58.961527    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9" podUID="9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c" containerName="main" containerID="cri-o://16a1b1df425f77da55a28e7dca8ed64d62013386ab417ffbc50b7f51a128ef08" gracePeriod=45
Sep 19 11:02:58.981265 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:58.981230    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc" podUID="df00f0e9-db3d-4496-bbf3-5156402deab9" containerName="main" containerID="cri-o://adcbef3de067badf930edbd5151e117726f1874e3c5b4abc648515c96d138622" gracePeriod=45
Sep 19 11:02:58.987099 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:58.987062    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76" podUID="19ae907f-14b0-49f5-95c2-bededfa382a4" containerName="main" containerID="cri-o://fb1f590b9b994782abaaf675f7262d4be018c0ca623770cd6ae680da9b7e1edd" gracePeriod=45
Sep 19 11:02:58.995980 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:58.995943    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6" podUID="10945f3e-0a8b-413e-b27c-76b1255e1704" containerName="main" containerID="cri-o://1cb25cb054c356697c8f51375b7913f9e67b9fc70b699f4b8ace66b880c9d6bf" gracePeriod=45
Sep 19 11:02:59.130545 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.130498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:02:59.294588 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.294006    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 11:02:59.296617 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.296588    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:02:59.325628 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.325606    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 11:02:59.334657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.334516    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 11:02:59.415557 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415518    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-model-cache\") pod \"df00f0e9-db3d-4496-bbf3-5156402deab9\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") "
Sep 19 11:02:59.415769 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415571    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6bpv\" (UniqueName: \"kubernetes.io/projected/df00f0e9-db3d-4496-bbf3-5156402deab9-kube-api-access-g6bpv\") pod \"df00f0e9-db3d-4496-bbf3-5156402deab9\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") "
Sep 19 11:02:59.415769 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-tmp-dir\") pod \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") "
Sep 19 11:02:59.415769 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415633    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-dshm\") pod \"19ae907f-14b0-49f5-95c2-bededfa382a4\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") "
Sep 19 11:02:59.415769 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415663    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-tmp-dir\") pod \"19ae907f-14b0-49f5-95c2-bededfa382a4\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") "
Sep 19 11:02:59.415769 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415694    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df00f0e9-db3d-4496-bbf3-5156402deab9-tls-certs\") pod \"df00f0e9-db3d-4496-bbf3-5156402deab9\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") "
Sep 19 11:02:59.415769 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415732    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-dshm\") pod \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") "
Sep 19 11:02:59.415769 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415771    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-dshm\") pod \"10945f3e-0a8b-413e-b27c-76b1255e1704\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") "
Sep 19 11:02:59.416135 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415802    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-dshm\") pod \"df00f0e9-db3d-4496-bbf3-5156402deab9\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") "
Sep 19 11:02:59.416135 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415830    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-model-cache\") pod \"10945f3e-0a8b-413e-b27c-76b1255e1704\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") "
Sep 19 11:02:59.416135 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415855    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-model-cache\") pod \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") "
Sep 19 11:02:59.416135 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415849    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-model-cache" (OuterVolumeSpecName: "model-cache") pod "df00f0e9-db3d-4496-bbf3-5156402deab9" (UID: "df00f0e9-db3d-4496-bbf3-5156402deab9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.416135 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415918    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-home\") pod \"10945f3e-0a8b-413e-b27c-76b1255e1704\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") "
Sep 19 11:02:59.416135 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415974    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hq4f\" (UniqueName: \"kubernetes.io/projected/10945f3e-0a8b-413e-b27c-76b1255e1704-kube-api-access-2hq4f\") pod \"10945f3e-0a8b-413e-b27c-76b1255e1704\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") "
Sep 19 11:02:59.416135 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416011    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7bqq\" (UniqueName: \"kubernetes.io/projected/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-kube-api-access-m7bqq\") pod \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") "
Sep 19 11:02:59.416135 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416070    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-model-cache\") pod \"19ae907f-14b0-49f5-95c2-bededfa382a4\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") "
Sep 19 11:02:59.416135 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416092    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c" (UID: "9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.416135 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416116    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwl2m\" (UniqueName: \"kubernetes.io/projected/19ae907f-14b0-49f5-95c2-bededfa382a4-kube-api-access-mwl2m\") pod \"19ae907f-14b0-49f5-95c2-bededfa382a4\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") "
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416144    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10945f3e-0a8b-413e-b27c-76b1255e1704-tls-certs\") pod \"10945f3e-0a8b-413e-b27c-76b1255e1704\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") "
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416178    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-tmp-dir\") pod \"10945f3e-0a8b-413e-b27c-76b1255e1704\" (UID: \"10945f3e-0a8b-413e-b27c-76b1255e1704\") "
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416220    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-home\") pod \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") "
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416242    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-tmp-dir\") pod \"df00f0e9-db3d-4496-bbf3-5156402deab9\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") "
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416259    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19ae907f-14b0-49f5-95c2-bededfa382a4" (UID: "19ae907f-14b0-49f5-95c2-bededfa382a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416288    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-model-cache" (OuterVolumeSpecName: "model-cache") pod "10945f3e-0a8b-413e-b27c-76b1255e1704" (UID: "10945f3e-0a8b-413e-b27c-76b1255e1704"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416284    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19ae907f-14b0-49f5-95c2-bededfa382a4-tls-certs\") pod \"19ae907f-14b0-49f5-95c2-bededfa382a4\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") "
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416345    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-home\") pod \"19ae907f-14b0-49f5-95c2-bededfa382a4\" (UID: \"19ae907f-14b0-49f5-95c2-bededfa382a4\") "
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416368    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-tls-certs\") pod \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\" (UID: \"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c\") "
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416416    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "19ae907f-14b0-49f5-95c2-bededfa382a4" (UID: "19ae907f-14b0-49f5-95c2-bededfa382a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.415849    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c" (UID: "9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-home\") pod \"df00f0e9-db3d-4496-bbf3-5156402deab9\" (UID: \"df00f0e9-db3d-4496-bbf3-5156402deab9\") "
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416573    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-home" (OuterVolumeSpecName: "home") pod "10945f3e-0a8b-413e-b27c-76b1255e1704" (UID: "10945f3e-0a8b-413e-b27c-76b1255e1704"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416936    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416955    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416969    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416985    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416990    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-home" (OuterVolumeSpecName: "home") pod "9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c" (UID: "9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.417013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.416999    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.418013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.417030    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.418013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.417046    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.418013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.417181    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10945f3e-0a8b-413e-b27c-76b1255e1704" (UID: "10945f3e-0a8b-413e-b27c-76b1255e1704"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.418013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.417722    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-home" (OuterVolumeSpecName: "home") pod "19ae907f-14b0-49f5-95c2-bededfa382a4" (UID: "19ae907f-14b0-49f5-95c2-bededfa382a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.418425 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.418395    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df00f0e9-db3d-4496-bbf3-5156402deab9" (UID: "df00f0e9-db3d-4496-bbf3-5156402deab9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.418495 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.418456    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-home" (OuterVolumeSpecName: "home") pod "df00f0e9-db3d-4496-bbf3-5156402deab9" (UID: "df00f0e9-db3d-4496-bbf3-5156402deab9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.419205 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.419175    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df00f0e9-db3d-4496-bbf3-5156402deab9-kube-api-access-g6bpv" (OuterVolumeSpecName: "kube-api-access-g6bpv") pod "df00f0e9-db3d-4496-bbf3-5156402deab9" (UID: "df00f0e9-db3d-4496-bbf3-5156402deab9"). InnerVolumeSpecName "kube-api-access-g6bpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:02:59.420644 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.420601    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19ae907f-14b0-49f5-95c2-bededfa382a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19ae907f-14b0-49f5-95c2-bededfa382a4" (UID: "19ae907f-14b0-49f5-95c2-bededfa382a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:02:59.420644 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.420619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-dshm" (OuterVolumeSpecName: "dshm") pod "9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c" (UID: "9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.421063 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.420011    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ae907f-14b0-49f5-95c2-bededfa382a4-kube-api-access-mwl2m" (OuterVolumeSpecName: "kube-api-access-mwl2m") pod "19ae907f-14b0-49f5-95c2-bededfa382a4" (UID: "19ae907f-14b0-49f5-95c2-bededfa382a4"). InnerVolumeSpecName "kube-api-access-mwl2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:02:59.421063 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.420746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-dshm" (OuterVolumeSpecName: "dshm") pod "df00f0e9-db3d-4496-bbf3-5156402deab9" (UID: "df00f0e9-db3d-4496-bbf3-5156402deab9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.421063 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.420788    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df00f0e9-db3d-4496-bbf3-5156402deab9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df00f0e9-db3d-4496-bbf3-5156402deab9" (UID: "df00f0e9-db3d-4496-bbf3-5156402deab9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:02:59.421063 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.420942    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-dshm" (OuterVolumeSpecName: "dshm") pod "19ae907f-14b0-49f5-95c2-bededfa382a4" (UID: "19ae907f-14b0-49f5-95c2-bededfa382a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.421517 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.421468    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10945f3e-0a8b-413e-b27c-76b1255e1704-kube-api-access-2hq4f" (OuterVolumeSpecName: "kube-api-access-2hq4f") pod "10945f3e-0a8b-413e-b27c-76b1255e1704" (UID: "10945f3e-0a8b-413e-b27c-76b1255e1704"). InnerVolumeSpecName "kube-api-access-2hq4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:02:59.421853 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.421825    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-kube-api-access-m7bqq" (OuterVolumeSpecName: "kube-api-access-m7bqq") pod "9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c" (UID: "9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c"). InnerVolumeSpecName "kube-api-access-m7bqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:02:59.421963 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.421892    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-dshm" (OuterVolumeSpecName: "dshm") pod "10945f3e-0a8b-413e-b27c-76b1255e1704" (UID: "10945f3e-0a8b-413e-b27c-76b1255e1704"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:02:59.422103 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.422075    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c" (UID: "9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:02:59.422353 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.422338    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10945f3e-0a8b-413e-b27c-76b1255e1704-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10945f3e-0a8b-413e-b27c-76b1255e1704" (UID: "10945f3e-0a8b-413e-b27c-76b1255e1704"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:02:59.517966 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.517855    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.517966 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.517893    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.517966 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.517905    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.517966 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.517916    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hq4f\" (UniqueName: \"kubernetes.io/projected/10945f3e-0a8b-413e-b27c-76b1255e1704-kube-api-access-2hq4f\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.517966 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.517931    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7bqq\" (UniqueName: \"kubernetes.io/projected/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-kube-api-access-m7bqq\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.517966 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.517946    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwl2m\" (UniqueName: \"kubernetes.io/projected/19ae907f-14b0-49f5-95c2-bededfa382a4-kube-api-access-mwl2m\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.517966 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.517960    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10945f3e-0a8b-413e-b27c-76b1255e1704-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.517966 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.517972    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10945f3e-0a8b-413e-b27c-76b1255e1704-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.518351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.517984    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.518351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.517994    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.518351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.518003    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19ae907f-14b0-49f5-95c2-bededfa382a4-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.518351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.518013    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.518351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.518022    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.518351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.518031    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df00f0e9-db3d-4496-bbf3-5156402deab9-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.518351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.518042    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6bpv\" (UniqueName: \"kubernetes.io/projected/df00f0e9-db3d-4496-bbf3-5156402deab9-kube-api-access-g6bpv\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.518351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.518052    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19ae907f-14b0-49f5-95c2-bededfa382a4-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:02:59.518351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:02:59.518063    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df00f0e9-db3d-4496-bbf3-5156402deab9-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:00.047852 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.047820    2580 generic.go:356] "Generic (PLEG): container finished" podID="10945f3e-0a8b-413e-b27c-76b1255e1704" containerID="1cb25cb054c356697c8f51375b7913f9e67b9fc70b699f4b8ace66b880c9d6bf" exitCode=0
Sep 19 11:03:00.048323 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.047867    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"
Sep 19 11:03:00.048323 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.047906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6" event={"ID":"10945f3e-0a8b-413e-b27c-76b1255e1704","Type":"ContainerDied","Data":"1cb25cb054c356697c8f51375b7913f9e67b9fc70b699f4b8ace66b880c9d6bf"}
Sep 19 11:03:00.048323 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.047951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6" event={"ID":"10945f3e-0a8b-413e-b27c-76b1255e1704","Type":"ContainerDied","Data":"d0e2a9c5bfadcaaab52ef9601084a6fce386d6324b37eaebe38d1e6ca658b283"}
Sep 19 11:03:00.048323 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.047969    2580 scope.go:117] "RemoveContainer" containerID="1cb25cb054c356697c8f51375b7913f9e67b9fc70b699f4b8ace66b880c9d6bf"
Sep 19 11:03:00.049523 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.049496    2580 generic.go:356] "Generic (PLEG): container finished" podID="19ae907f-14b0-49f5-95c2-bededfa382a4" containerID="fb1f590b9b994782abaaf675f7262d4be018c0ca623770cd6ae680da9b7e1edd" exitCode=0
Sep 19 11:03:00.049643 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.049531    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"
Sep 19 11:03:00.049643 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.049586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76" event={"ID":"19ae907f-14b0-49f5-95c2-bededfa382a4","Type":"ContainerDied","Data":"fb1f590b9b994782abaaf675f7262d4be018c0ca623770cd6ae680da9b7e1edd"}
Sep 19 11:03:00.049643 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.049615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76" event={"ID":"19ae907f-14b0-49f5-95c2-bededfa382a4","Type":"ContainerDied","Data":"82c23b4e342d3d96cf08ee47315fdfb8a1c11a0caba9263443acb83a861079fb"}
Sep 19 11:03:00.051191 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.051166    2580 generic.go:356] "Generic (PLEG): container finished" podID="df00f0e9-db3d-4496-bbf3-5156402deab9" containerID="adcbef3de067badf930edbd5151e117726f1874e3c5b4abc648515c96d138622" exitCode=0
Sep 19 11:03:00.051301 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.051278    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"
Sep 19 11:03:00.051368 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.051291    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc" event={"ID":"df00f0e9-db3d-4496-bbf3-5156402deab9","Type":"ContainerDied","Data":"adcbef3de067badf930edbd5151e117726f1874e3c5b4abc648515c96d138622"}
Sep 19 11:03:00.051705 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.051676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc" event={"ID":"df00f0e9-db3d-4496-bbf3-5156402deab9","Type":"ContainerDied","Data":"f11b4627ca9d6ff4620c3291294e5b16743d66bb9d6f6870d81474a8bb56b706"}
Sep 19 11:03:00.053303 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.053273    2580 generic.go:356] "Generic (PLEG): container finished" podID="9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c" containerID="16a1b1df425f77da55a28e7dca8ed64d62013386ab417ffbc50b7f51a128ef08" exitCode=0
Sep 19 11:03:00.053402 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.053303    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"
Sep 19 11:03:00.053402 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.053331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9" event={"ID":"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c","Type":"ContainerDied","Data":"16a1b1df425f77da55a28e7dca8ed64d62013386ab417ffbc50b7f51a128ef08"}
Sep 19 11:03:00.053402 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.053358    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9" event={"ID":"9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c","Type":"ContainerDied","Data":"84f35837a3bb6bc064b28a926dd0fdac86b7dec54e7f23e6d7828f1018bd55e2"}
Sep 19 11:03:00.058403 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.058386    2580 scope.go:117] "RemoveContainer" containerID="1cb25cb054c356697c8f51375b7913f9e67b9fc70b699f4b8ace66b880c9d6bf"
Sep 19 11:03:00.058736 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:03:00.058718    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cb25cb054c356697c8f51375b7913f9e67b9fc70b699f4b8ace66b880c9d6bf\": container with ID starting with 1cb25cb054c356697c8f51375b7913f9e67b9fc70b699f4b8ace66b880c9d6bf not found: ID does not exist" containerID="1cb25cb054c356697c8f51375b7913f9e67b9fc70b699f4b8ace66b880c9d6bf"
Sep 19 11:03:00.058824 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.058745    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb25cb054c356697c8f51375b7913f9e67b9fc70b699f4b8ace66b880c9d6bf"} err="failed to get container status \"1cb25cb054c356697c8f51375b7913f9e67b9fc70b699f4b8ace66b880c9d6bf\": rpc error: code = NotFound desc = could not find container \"1cb25cb054c356697c8f51375b7913f9e67b9fc70b699f4b8ace66b880c9d6bf\": container with ID starting with 1cb25cb054c356697c8f51375b7913f9e67b9fc70b699f4b8ace66b880c9d6bf not found: ID does not exist"
Sep 19 11:03:00.058824 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.058781    2580 scope.go:117] "RemoveContainer" containerID="fb1f590b9b994782abaaf675f7262d4be018c0ca623770cd6ae680da9b7e1edd"
Sep 19 11:03:00.068292 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.068272    2580 scope.go:117] "RemoveContainer" containerID="fb1f590b9b994782abaaf675f7262d4be018c0ca623770cd6ae680da9b7e1edd"
Sep 19 11:03:00.068691 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:03:00.068650    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1f590b9b994782abaaf675f7262d4be018c0ca623770cd6ae680da9b7e1edd\": container with ID starting with fb1f590b9b994782abaaf675f7262d4be018c0ca623770cd6ae680da9b7e1edd not found: ID does not exist" containerID="fb1f590b9b994782abaaf675f7262d4be018c0ca623770cd6ae680da9b7e1edd"
Sep 19 11:03:00.068826 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.068701    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1f590b9b994782abaaf675f7262d4be018c0ca623770cd6ae680da9b7e1edd"} err="failed to get container status \"fb1f590b9b994782abaaf675f7262d4be018c0ca623770cd6ae680da9b7e1edd\": rpc error: code = NotFound desc = could not find container \"fb1f590b9b994782abaaf675f7262d4be018c0ca623770cd6ae680da9b7e1edd\": container with ID starting with fb1f590b9b994782abaaf675f7262d4be018c0ca623770cd6ae680da9b7e1edd not found: ID does not exist"
Sep 19 11:03:00.068826 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.068736    2580 scope.go:117] "RemoveContainer" containerID="adcbef3de067badf930edbd5151e117726f1874e3c5b4abc648515c96d138622"
Sep 19 11:03:00.070161 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.070142    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"]
Sep 19 11:03:00.071966 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.071945    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/simulator-w0-1907c1-kserve-7f8b595564-h7gk6"]
Sep 19 11:03:00.077822 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.077805    2580 scope.go:117] "RemoveContainer" containerID="adcbef3de067badf930edbd5151e117726f1874e3c5b4abc648515c96d138622"
Sep 19 11:03:00.078237 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:03:00.078209    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcbef3de067badf930edbd5151e117726f1874e3c5b4abc648515c96d138622\": container with ID starting with adcbef3de067badf930edbd5151e117726f1874e3c5b4abc648515c96d138622 not found: ID does not exist" containerID="adcbef3de067badf930edbd5151e117726f1874e3c5b4abc648515c96d138622"
Sep 19 11:03:00.078351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.078247    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcbef3de067badf930edbd5151e117726f1874e3c5b4abc648515c96d138622"} err="failed to get container status \"adcbef3de067badf930edbd5151e117726f1874e3c5b4abc648515c96d138622\": rpc error: code = NotFound desc = could not find container \"adcbef3de067badf930edbd5151e117726f1874e3c5b4abc648515c96d138622\": container with ID starting with adcbef3de067badf930edbd5151e117726f1874e3c5b4abc648515c96d138622 not found: ID does not exist"
Sep 19 11:03:00.078351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.078271    2580 scope.go:117] "RemoveContainer" containerID="16a1b1df425f77da55a28e7dca8ed64d62013386ab417ffbc50b7f51a128ef08"
Sep 19 11:03:00.080850 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.080830    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"]
Sep 19 11:03:00.086439 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.086416    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/embedding-w0-1907c1-kserve-75f7998d69-bwxxc"]
Sep 19 11:03:00.087512 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.087495    2580 scope.go:117] "RemoveContainer" containerID="16a1b1df425f77da55a28e7dca8ed64d62013386ab417ffbc50b7f51a128ef08"
Sep 19 11:03:00.087837 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:03:00.087819    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a1b1df425f77da55a28e7dca8ed64d62013386ab417ffbc50b7f51a128ef08\": container with ID starting with 16a1b1df425f77da55a28e7dca8ed64d62013386ab417ffbc50b7f51a128ef08 not found: ID does not exist" containerID="16a1b1df425f77da55a28e7dca8ed64d62013386ab417ffbc50b7f51a128ef08"
Sep 19 11:03:00.087886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.087847    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a1b1df425f77da55a28e7dca8ed64d62013386ab417ffbc50b7f51a128ef08"} err="failed to get container status \"16a1b1df425f77da55a28e7dca8ed64d62013386ab417ffbc50b7f51a128ef08\": rpc error: code = NotFound desc = could not find container \"16a1b1df425f77da55a28e7dca8ed64d62013386ab417ffbc50b7f51a128ef08\": container with ID starting with 16a1b1df425f77da55a28e7dca8ed64d62013386ab417ffbc50b7f51a128ef08 not found: ID does not exist"
Sep 19 11:03:00.095030 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.095008    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"]
Sep 19 11:03:00.098595 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.098576    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/premium-w0-1907c1-kserve-6c774554f7-g4v76"]
Sep 19 11:03:00.108326 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.108201    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"]
Sep 19 11:03:00.109784 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:00.109766    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9"]
Sep 19 11:03:01.174843 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:01.174815    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:01.275916 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:01.275883    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:01.425518 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:01.425422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:01.856523 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:01.856490    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10945f3e-0a8b-413e-b27c-76b1255e1704" path="/var/lib/kubelet/pods/10945f3e-0a8b-413e-b27c-76b1255e1704/volumes"
Sep 19 11:03:01.856995 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:01.856976    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19ae907f-14b0-49f5-95c2-bededfa382a4" path="/var/lib/kubelet/pods/19ae907f-14b0-49f5-95c2-bededfa382a4/volumes"
Sep 19 11:03:01.857384 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:01.857366    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c" path="/var/lib/kubelet/pods/9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c/volumes"
Sep 19 11:03:01.858366 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:01.858337    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df00f0e9-db3d-4496-bbf3-5156402deab9" path="/var/lib/kubelet/pods/df00f0e9-db3d-4496-bbf3-5156402deab9/volumes"
Sep 19 11:03:04.242338 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:04.242278    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1907c1/distinct-2-w0-1907c1-kserve-7b667b77b7-vbzt9" podUID="9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 19 11:03:04.716613 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:04.716571    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:04.815884 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:04.815848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:04.903366 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:04.903326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:04.958123 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:04.958079    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:05.233529 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:05.233495    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:05.403160 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:05.403129    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:06.040225 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:06.040193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:06.829180 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:06.829151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:08.898087 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:08.898050    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr" podUID="fb65cc2f-1fda-4c77-b076-7e4dd789bfdf" containerName="main" containerID="cri-o://70044462fd73d3060fd7c084663a10bff25d3b3f2609584449b3f5bb53b216c1" gracePeriod=45
Sep 19 11:03:08.916261 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:08.916231    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59" podUID="c47630c3-6941-4929-8c74-a79aa06e92a8" containerName="main" containerID="cri-o://d786fd67157b566e6d69947fefb5744b91a70bfe8c7e39b196e5181f7cef8981" gracePeriod=45
Sep 19 11:03:09.105212 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.105181    2580 generic.go:356] "Generic (PLEG): container finished" podID="c47630c3-6941-4929-8c74-a79aa06e92a8" containerID="d786fd67157b566e6d69947fefb5744b91a70bfe8c7e39b196e5181f7cef8981" exitCode=0
Sep 19 11:03:09.105389 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.105256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59" event={"ID":"c47630c3-6941-4929-8c74-a79aa06e92a8","Type":"ContainerDied","Data":"d786fd67157b566e6d69947fefb5744b91a70bfe8c7e39b196e5181f7cef8981"}
Sep 19 11:03:09.107021 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.106997    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb65cc2f-1fda-4c77-b076-7e4dd789bfdf" containerID="70044462fd73d3060fd7c084663a10bff25d3b3f2609584449b3f5bb53b216c1" exitCode=0
Sep 19 11:03:09.107145 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.107033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr" event={"ID":"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf","Type":"ContainerDied","Data":"70044462fd73d3060fd7c084663a10bff25d3b3f2609584449b3f5bb53b216c1"}
Sep 19 11:03:09.195698 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.195662    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 11:03:09.200732 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.200703    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 11:03:09.304050 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304016    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-tmp-dir\") pod \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") "
Sep 19 11:03:09.304267 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304073    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-tmp-dir\") pod \"c47630c3-6941-4929-8c74-a79aa06e92a8\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") "
Sep 19 11:03:09.304267 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-dshm\") pod \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") "
Sep 19 11:03:09.304267 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304126    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7psxl\" (UniqueName: \"kubernetes.io/projected/c47630c3-6941-4929-8c74-a79aa06e92a8-kube-api-access-7psxl\") pod \"c47630c3-6941-4929-8c74-a79aa06e92a8\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") "
Sep 19 11:03:09.304267 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304143    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljgw5\" (UniqueName: \"kubernetes.io/projected/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-kube-api-access-ljgw5\") pod \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") "
Sep 19 11:03:09.304267 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304167    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-model-cache\") pod \"c47630c3-6941-4929-8c74-a79aa06e92a8\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") "
Sep 19 11:03:09.304267 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304208    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-home\") pod \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") "
Sep 19 11:03:09.304267 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304256    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-model-cache\") pod \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") "
Sep 19 11:03:09.304637 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304294    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-tls-certs\") pod \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\" (UID: \"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf\") "
Sep 19 11:03:09.304637 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304329    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-home\") pod \"c47630c3-6941-4929-8c74-a79aa06e92a8\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") "
Sep 19 11:03:09.304637 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304361    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-dshm\") pod \"c47630c3-6941-4929-8c74-a79aa06e92a8\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") "
Sep 19 11:03:09.304637 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304355    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb65cc2f-1fda-4c77-b076-7e4dd789bfdf" (UID: "fb65cc2f-1fda-4c77-b076-7e4dd789bfdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:03:09.304637 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47630c3-6941-4929-8c74-a79aa06e92a8-tls-certs\") pod \"c47630c3-6941-4929-8c74-a79aa06e92a8\" (UID: \"c47630c3-6941-4929-8c74-a79aa06e92a8\") "
Sep 19 11:03:09.304637 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304358    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c47630c3-6941-4929-8c74-a79aa06e92a8" (UID: "c47630c3-6941-4929-8c74-a79aa06e92a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:03:09.304637 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304490    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "c47630c3-6941-4929-8c74-a79aa06e92a8" (UID: "c47630c3-6941-4929-8c74-a79aa06e92a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:03:09.304637 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304590    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-home" (OuterVolumeSpecName: "home") pod "fb65cc2f-1fda-4c77-b076-7e4dd789bfdf" (UID: "fb65cc2f-1fda-4c77-b076-7e4dd789bfdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:03:09.305091 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304651    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:09.305091 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304669    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:09.305091 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304683    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:09.305091 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304695    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:09.305091 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304713    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-home" (OuterVolumeSpecName: "home") pod "c47630c3-6941-4929-8c74-a79aa06e92a8" (UID: "c47630c3-6941-4929-8c74-a79aa06e92a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:03:09.305091 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.304985    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb65cc2f-1fda-4c77-b076-7e4dd789bfdf" (UID: "fb65cc2f-1fda-4c77-b076-7e4dd789bfdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:03:09.306984 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.306962    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-dshm" (OuterVolumeSpecName: "dshm") pod "fb65cc2f-1fda-4c77-b076-7e4dd789bfdf" (UID: "fb65cc2f-1fda-4c77-b076-7e4dd789bfdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:03:09.307192 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.307169    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb65cc2f-1fda-4c77-b076-7e4dd789bfdf" (UID: "fb65cc2f-1fda-4c77-b076-7e4dd789bfdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:03:09.307238 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.307187    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47630c3-6941-4929-8c74-a79aa06e92a8-kube-api-access-7psxl" (OuterVolumeSpecName: "kube-api-access-7psxl") pod "c47630c3-6941-4929-8c74-a79aa06e92a8" (UID: "c47630c3-6941-4929-8c74-a79aa06e92a8"). InnerVolumeSpecName "kube-api-access-7psxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:03:09.307375 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.307357    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-dshm" (OuterVolumeSpecName: "dshm") pod "c47630c3-6941-4929-8c74-a79aa06e92a8" (UID: "c47630c3-6941-4929-8c74-a79aa06e92a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:03:09.307375 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.307372    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c47630c3-6941-4929-8c74-a79aa06e92a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c47630c3-6941-4929-8c74-a79aa06e92a8" (UID: "c47630c3-6941-4929-8c74-a79aa06e92a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:03:09.307488 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.307379    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-kube-api-access-ljgw5" (OuterVolumeSpecName: "kube-api-access-ljgw5") pod "fb65cc2f-1fda-4c77-b076-7e4dd789bfdf" (UID: "fb65cc2f-1fda-4c77-b076-7e4dd789bfdf"). InnerVolumeSpecName "kube-api-access-ljgw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:03:09.406104 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.406052    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:09.406104 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.406100    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7psxl\" (UniqueName: \"kubernetes.io/projected/c47630c3-6941-4929-8c74-a79aa06e92a8-kube-api-access-7psxl\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:09.406104 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.406112    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljgw5\" (UniqueName: \"kubernetes.io/projected/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-kube-api-access-ljgw5\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:09.406104 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.406121    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:09.406394 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.406129    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:09.406394 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.406137    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:09.406394 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.406144    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47630c3-6941-4929-8c74-a79aa06e92a8-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:09.406394 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:09.406151    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47630c3-6941-4929-8c74-a79aa06e92a8-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:10.113897 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:10.113871    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"
Sep 19 11:03:10.114351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:10.113906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr" event={"ID":"fb65cc2f-1fda-4c77-b076-7e4dd789bfdf","Type":"ContainerDied","Data":"13848fbcbb968ee21ff1697cdea5fda44b406d9d97504d69e96c165e15b9cfb7"}
Sep 19 11:03:10.114351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:10.113953    2580 scope.go:117] "RemoveContainer" containerID="70044462fd73d3060fd7c084663a10bff25d3b3f2609584449b3f5bb53b216c1"
Sep 19 11:03:10.115486 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:10.115464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59" event={"ID":"c47630c3-6941-4929-8c74-a79aa06e92a8","Type":"ContainerDied","Data":"1539ba59740558c31ad41c18f0eb69dabeae48e53568467dcf3f162562008c1b"}
Sep 19 11:03:10.115580 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:10.115478    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"
Sep 19 11:03:10.122689 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:10.122672    2580 scope.go:117] "RemoveContainer" containerID="d786fd67157b566e6d69947fefb5744b91a70bfe8c7e39b196e5181f7cef8981"
Sep 19 11:03:10.131557 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:10.131536    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"]
Sep 19 11:03:10.133135 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:10.133110    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/simulator-w4-5116ec-kserve-7c6597457d-vpz59"]
Sep 19 11:03:10.141436 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:10.141414    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"]
Sep 19 11:03:10.144804 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:10.144783    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5116ec/distinct-w4-5116ec-kserve-dcb7456f8-z6knr"]
Sep 19 11:03:10.978430 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:10.978399    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:11.078445 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:11.078413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:11.234890 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:11.234807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:11.440683 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:11.440649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:11.857068 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:11.857036    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c47630c3-6941-4929-8c74-a79aa06e92a8" path="/var/lib/kubelet/pods/c47630c3-6941-4929-8c74-a79aa06e92a8/volumes"
Sep 19 11:03:11.857426 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:11.857413    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb65cc2f-1fda-4c77-b076-7e4dd789bfdf" path="/var/lib/kubelet/pods/fb65cc2f-1fda-4c77-b076-7e4dd789bfdf/volumes"
Sep 19 11:03:18.405485 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:18.405447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:03:21.172170 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:21.172132    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:21.271680 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:21.271648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:21.592693 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:21.592646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:27.077337 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:27.077298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:30.798437 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:30.798400    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:30.850062 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:30.850031    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:31.190093 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:31.190045    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:34.007364 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:34.007326    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:03:36.720298 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:36.720257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:39.253269 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:39.253236    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"]
Sep 19 11:03:39.271487 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:39.271455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:40.251581 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:40.251532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:48.617853 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:48.617820    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:03:54.275197 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.275131    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6" podUID="9cd5c909-ef48-41d3-85dc-c1b9bd0fd953" containerName="main" containerID="cri-o://e39589247cfb301883826c8cbb57127898ab72a5b8c071975eebdc602f90358d" gracePeriod=45
Sep 19 11:03:54.535782 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.535742    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 11:03:54.699009 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.698975    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-home\") pod \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") "
Sep 19 11:03:54.699209 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.699028    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-dshm\") pod \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") "
Sep 19 11:03:54.699209 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.699069    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptb9f\" (UniqueName: \"kubernetes.io/projected/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-kube-api-access-ptb9f\") pod \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") "
Sep 19 11:03:54.699209 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.699107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-tmp-dir\") pod \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") "
Sep 19 11:03:54.699209 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.699143    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-tls-certs\") pod \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") "
Sep 19 11:03:54.699209 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.699194    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-model-cache\") pod \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\" (UID: \"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953\") "
Sep 19 11:03:54.699495 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.699237    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-home" (OuterVolumeSpecName: "home") pod "9cd5c909-ef48-41d3-85dc-c1b9bd0fd953" (UID: "9cd5c909-ef48-41d3-85dc-c1b9bd0fd953"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:03:54.699495 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.699401    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9cd5c909-ef48-41d3-85dc-c1b9bd0fd953" (UID: "9cd5c909-ef48-41d3-85dc-c1b9bd0fd953"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:03:54.699495 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.699436    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:54.699602 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.699518    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-model-cache" (OuterVolumeSpecName: "model-cache") pod "9cd5c909-ef48-41d3-85dc-c1b9bd0fd953" (UID: "9cd5c909-ef48-41d3-85dc-c1b9bd0fd953"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:03:54.701648 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.701624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-dshm" (OuterVolumeSpecName: "dshm") pod "9cd5c909-ef48-41d3-85dc-c1b9bd0fd953" (UID: "9cd5c909-ef48-41d3-85dc-c1b9bd0fd953"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:03:54.701806 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.701638    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9cd5c909-ef48-41d3-85dc-c1b9bd0fd953" (UID: "9cd5c909-ef48-41d3-85dc-c1b9bd0fd953"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:03:54.701806 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.701649    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-kube-api-access-ptb9f" (OuterVolumeSpecName: "kube-api-access-ptb9f") pod "9cd5c909-ef48-41d3-85dc-c1b9bd0fd953" (UID: "9cd5c909-ef48-41d3-85dc-c1b9bd0fd953"). InnerVolumeSpecName "kube-api-access-ptb9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:03:54.800796 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.800690    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:54.800796 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.800719    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptb9f\" (UniqueName: \"kubernetes.io/projected/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-kube-api-access-ptb9f\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:54.800796 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.800729    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:54.800796 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.800737    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:54.800796 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:54.800745    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:03:55.345059 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.345027    2580 generic.go:356] "Generic (PLEG): container finished" podID="9cd5c909-ef48-41d3-85dc-c1b9bd0fd953" containerID="e39589247cfb301883826c8cbb57127898ab72a5b8c071975eebdc602f90358d" exitCode=0
Sep 19 11:03:55.345474 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.345067    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"
Sep 19 11:03:55.345474 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.345116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6" event={"ID":"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953","Type":"ContainerDied","Data":"e39589247cfb301883826c8cbb57127898ab72a5b8c071975eebdc602f90358d"}
Sep 19 11:03:55.345474 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.345166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6" event={"ID":"9cd5c909-ef48-41d3-85dc-c1b9bd0fd953","Type":"ContainerDied","Data":"981877723cd282772aa0f0905a043f13c18fff379dc0fd960a97bd579fd37437"}
Sep 19 11:03:55.345474 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.345183    2580 scope.go:117] "RemoveContainer" containerID="e39589247cfb301883826c8cbb57127898ab72a5b8c071975eebdc602f90358d"
Sep 19 11:03:55.354345 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.354329    2580 scope.go:117] "RemoveContainer" containerID="e39589247cfb301883826c8cbb57127898ab72a5b8c071975eebdc602f90358d"
Sep 19 11:03:55.354623 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:03:55.354606    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39589247cfb301883826c8cbb57127898ab72a5b8c071975eebdc602f90358d\": container with ID starting with e39589247cfb301883826c8cbb57127898ab72a5b8c071975eebdc602f90358d not found: ID does not exist" containerID="e39589247cfb301883826c8cbb57127898ab72a5b8c071975eebdc602f90358d"
Sep 19 11:03:55.354679 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.354634    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39589247cfb301883826c8cbb57127898ab72a5b8c071975eebdc602f90358d"} err="failed to get container status \"e39589247cfb301883826c8cbb57127898ab72a5b8c071975eebdc602f90358d\": rpc error: code = NotFound desc = could not find container \"e39589247cfb301883826c8cbb57127898ab72a5b8c071975eebdc602f90358d\": container with ID starting with e39589247cfb301883826c8cbb57127898ab72a5b8c071975eebdc602f90358d not found: ID does not exist"
Sep 19 11:03:55.364858 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.364827    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"]
Sep 19 11:03:55.368452 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.368426    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/auth-test-model-137f88-kserve-cf66bc4bc-qkdd6"]
Sep 19 11:03:55.496444 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496405    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"]
Sep 19 11:03:55.496740 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496726    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89353aaf-f742-434b-9b75-f6fde2b8babc" containerName="maas-api"
Sep 19 11:03:55.496740 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496740    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89353aaf-f742-434b-9b75-f6fde2b8babc" containerName="maas-api"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496749    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47630c3-6941-4929-8c74-a79aa06e92a8" containerName="main"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496769    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47630c3-6941-4929-8c74-a79aa06e92a8" containerName="main"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496776    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10945f3e-0a8b-413e-b27c-76b1255e1704" containerName="main"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496781    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="10945f3e-0a8b-413e-b27c-76b1255e1704" containerName="main"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496792    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b01b51-bde4-46c3-b747-75f037aa78b5" containerName="payload-processing"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496798    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b01b51-bde4-46c3-b747-75f037aa78b5" containerName="payload-processing"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496804    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19ae907f-14b0-49f5-95c2-bededfa382a4" containerName="main"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496809    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19ae907f-14b0-49f5-95c2-bededfa382a4" containerName="main"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496815    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971fea77-05d6-4ce7-b005-85f93a5f727d" containerName="payload-pre-processing"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496821    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="971fea77-05d6-4ce7-b005-85f93a5f727d" containerName="payload-pre-processing"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496827    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c" containerName="main"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496831    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c" containerName="main"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496837    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df00f0e9-db3d-4496-bbf3-5156402deab9" containerName="main"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496843    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="df00f0e9-db3d-4496-bbf3-5156402deab9" containerName="main"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496854    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47f9f3a0-34f8-46dc-898c-459f5dd7004c" containerName="maas-api"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496859    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="47f9f3a0-34f8-46dc-898c-459f5dd7004c" containerName="maas-api"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496866    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195d93e0-1853-490e-8d78-e48fc35a4021" containerName="payload-processing"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496871    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="195d93e0-1853-490e-8d78-e48fc35a4021" containerName="payload-processing"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496882    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb65cc2f-1fda-4c77-b076-7e4dd789bfdf" containerName="main"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496887    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb65cc2f-1fda-4c77-b076-7e4dd789bfdf" containerName="main"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496892    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd5c909-ef48-41d3-85dc-c1b9bd0fd953" containerName="main"
Sep 19 11:03:55.496886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496897    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd5c909-ef48-41d3-85dc-c1b9bd0fd953" containerName="main"
Sep 19 11:03:55.498026 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496960    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="195d93e0-1853-490e-8d78-e48fc35a4021" containerName="payload-processing"
Sep 19 11:03:55.498026 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496967    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="47f9f3a0-34f8-46dc-898c-459f5dd7004c" containerName="maas-api"
Sep 19 11:03:55.498026 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496973    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="89353aaf-f742-434b-9b75-f6fde2b8babc" containerName="maas-api"
Sep 19 11:03:55.498026 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496978    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e0a60ed-75c3-4c9c-a759-5bc9b46ef45c" containerName="main"
Sep 19 11:03:55.498026 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496984    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="df00f0e9-db3d-4496-bbf3-5156402deab9" containerName="main"
Sep 19 11:03:55.498026 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496988    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47630c3-6941-4929-8c74-a79aa06e92a8" containerName="main"
Sep 19 11:03:55.498026 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.496995    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="10945f3e-0a8b-413e-b27c-76b1255e1704" containerName="main"
Sep 19 11:03:55.498026 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.497000    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19ae907f-14b0-49f5-95c2-bededfa382a4" containerName="main"
Sep 19 11:03:55.498026 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.497004    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd5c909-ef48-41d3-85dc-c1b9bd0fd953" containerName="main"
Sep 19 11:03:55.498026 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.497009    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb65cc2f-1fda-4c77-b076-7e4dd789bfdf" containerName="main"
Sep 19 11:03:55.498026 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.497013    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="971fea77-05d6-4ce7-b005-85f93a5f727d" containerName="payload-pre-processing"
Sep 19 11:03:55.498026 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.497019    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="63b01b51-bde4-46c3-b747-75f037aa78b5" containerName="payload-processing"
Sep 19 11:03:55.502034 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.502011    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.504430 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.504393    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-137f88\"/\"openshift-service-ca.crt\""
Sep 19 11:03:55.504572 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.504464    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-137f88\"/\"kube-root-ca.crt\""
Sep 19 11:03:55.504572 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.504562    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-137f88\"/\"default-dockercfg-986l4\""
Sep 19 11:03:55.504702 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.504646    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-137f88\"/\"sub-test-model-137f88-kserve-self-signed-certs\""
Sep 19 11:03:55.510348 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.509841    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"]
Sep 19 11:03:55.529219 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.529179    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:03:55.608069 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.607970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-tmp-dir\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.608069 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.608013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-dshm\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.608069 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.608040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbvgj\" (UniqueName: \"kubernetes.io/projected/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-kube-api-access-gbvgj\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.608295 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.608077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-model-cache\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.608295 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.608118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-home\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.608295 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.608144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-tls-certs\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.709129 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.709086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-tmp-dir\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.709129 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.709128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-dshm\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.709392 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.709144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbvgj\" (UniqueName: \"kubernetes.io/projected/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-kube-api-access-gbvgj\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.709392 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.709172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-model-cache\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.709392 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.709207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-home\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.709392 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.709228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-tls-certs\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.709566 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.709480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-tmp-dir\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.709639 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.709619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-home\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.709680 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.709641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-model-cache\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.711577 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.711557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-dshm\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.711833 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.711816    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-tls-certs\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.716392 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.716369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbvgj\" (UniqueName: \"kubernetes.io/projected/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-kube-api-access-gbvgj\") pod \"sub-test-model-137f88-kserve-5b6f974545-rckn4\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") " pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.814962 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.814930    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-137f88\"/\"default-dockercfg-986l4\""
Sep 19 11:03:55.823595 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.823573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:03:55.856619 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:55.856583    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd5c909-ef48-41d3-85dc-c1b9bd0fd953" path="/var/lib/kubelet/pods/9cd5c909-ef48-41d3-85dc-c1b9bd0fd953/volumes"
Sep 19 11:03:56.096008 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:56.095983    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"]
Sep 19 11:03:56.100547 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:03:56.100527    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c7452b1_28f3_4ffb_8d87_6b3b2d06efa9.slice/crio-3d9d38bef0705537fa64b40114ca5278ae7956f3aec89194c271a84d06ba2efa WatchSource:0}: Error finding container 3d9d38bef0705537fa64b40114ca5278ae7956f3aec89194c271a84d06ba2efa: Status 404 returned error can't find the container with id 3d9d38bef0705537fa64b40114ca5278ae7956f3aec89194c271a84d06ba2efa
Sep 19 11:03:56.351686 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:56.351655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4" event={"ID":"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9","Type":"ContainerStarted","Data":"110733ff4bc116f8b94f6d7da7351d2a0ed382fb145605ee88656d3588d4ca93"}
Sep 19 11:03:56.351686 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:56.351692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4" event={"ID":"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9","Type":"ContainerStarted","Data":"3d9d38bef0705537fa64b40114ca5278ae7956f3aec89194c271a84d06ba2efa"}
Sep 19 11:03:56.369568 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:56.369515    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4" podStartSLOduration=1.36949675 podStartE2EDuration="1.36949675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:03:56.367434705 +0000 UTC m=+1681.026179862" watchObservedRunningTime="2026-09-19 11:03:56.36949675 +0000 UTC m=+1681.028241909"
Sep 19 11:03:59.109252 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:03:59.109209    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:04:05.824317 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:04:05.824275    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:04:05.824946 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:04:05.824342    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:04:05.836916 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:04:05.836882    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:04:06.414897 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:04:06.414865    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:04:09.739459 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:04:09.739427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:04:13.216308 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:04:13.216271    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:04:28.617137 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:04:28.617048    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:04:38.306334 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:04:38.306296    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:04:46.211456 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:04:46.211423    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"]
Sep 19 11:04:46.229048 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:04:46.229012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:04:47.215526 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:04:47.215496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:04:48.813893 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:04:48.813862    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:04:59.512702 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:04:59.512661    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:05:00.514035 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.514006    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"]
Sep 19 11:05:00.519508 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.519478    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.521741 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.521717    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-137f88\"/\"rate-test-model-137f88-kserve-self-signed-certs\""
Sep 19 11:05:00.527864 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.527841    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"]
Sep 19 11:05:00.539806 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.539772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:05:00.570950 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.570922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-dshm\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.571106 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.570958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-home\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.571106 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.571020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-model-cache\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.571106 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.571040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-tls-certs\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.571216 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.571093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626j8\" (UniqueName: \"kubernetes.io/projected/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-kube-api-access-626j8\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.571216 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.571131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-tmp-dir\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.671666 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.671628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-model-cache\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.671850 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.671671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-tls-certs\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.671850 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.671739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626j8\" (UniqueName: \"kubernetes.io/projected/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-kube-api-access-626j8\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.671850 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.671787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-tmp-dir\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.671850 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.671843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-dshm\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.672078 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.671951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-home\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.672078 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.672069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-model-cache\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.672187 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.672146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-tmp-dir\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.672264 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.672244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-home\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.674514 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.674489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-dshm\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.674624 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.674545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-tls-certs\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.679726 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.679705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-626j8\" (UniqueName: \"kubernetes.io/projected/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-kube-api-access-626j8\") pod \"rate-test-model-137f88-kserve-779c974dc9-l76ff\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") " pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:00.830482 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:00.830381    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:01.102414 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.102386    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"]
Sep 19 11:05:01.104585 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:05:01.104561    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca0f0c7e_6362_4223_a1dc_5e82108e4dbc.slice/crio-a8d2f1165234b4c9c0b0aaee89a1c745ba6f62b480234579ede1ed7b691163d0 WatchSource:0}: Error finding container a8d2f1165234b4c9c0b0aaee89a1c745ba6f62b480234579ede1ed7b691163d0: Status 404 returned error can't find the container with id a8d2f1165234b4c9c0b0aaee89a1c745ba6f62b480234579ede1ed7b691163d0
Sep 19 11:05:01.231533 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.231484    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4" podUID="2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9" containerName="main" containerID="cri-o://110733ff4bc116f8b94f6d7da7351d2a0ed382fb145605ee88656d3588d4ca93" gracePeriod=45
Sep 19 11:05:01.471322 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.471294    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:05:01.477635 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.477609    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-tls-certs\") pod \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") "
Sep 19 11:05:01.477819 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.477674    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-tmp-dir\") pod \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") "
Sep 19 11:05:01.477819 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.477691    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-home\") pod \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") "
Sep 19 11:05:01.477819 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.477708    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbvgj\" (UniqueName: \"kubernetes.io/projected/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-kube-api-access-gbvgj\") pod \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") "
Sep 19 11:05:01.477819 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.477730    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-model-cache\") pod \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") "
Sep 19 11:05:01.477819 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.477808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-dshm\") pod \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\" (UID: \"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9\") "
Sep 19 11:05:01.478094 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.477963    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-home" (OuterVolumeSpecName: "home") pod "2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9" (UID: "2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:05:01.478094 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.477978    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9" (UID: "2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:05:01.478212 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.478122    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:05:01.478212 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.478129    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9" (UID: "2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:05:01.478212 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.478138    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:05:01.479958 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.479934    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-dshm" (OuterVolumeSpecName: "dshm") pod "2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9" (UID: "2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:05:01.480053 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.479961    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9" (UID: "2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:05:01.480053 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.479972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-kube-api-access-gbvgj" (OuterVolumeSpecName: "kube-api-access-gbvgj") pod "2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9" (UID: "2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9"). InnerVolumeSpecName "kube-api-access-gbvgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:05:01.578790 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.578722    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbvgj\" (UniqueName: \"kubernetes.io/projected/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-kube-api-access-gbvgj\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:05:01.578790 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.578781    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:05:01.578790 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.578791    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:05:01.578790 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.578799    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:05:01.679796 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.679686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff" event={"ID":"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc","Type":"ContainerStarted","Data":"add2e60304344cf5d5ab206bd6b65ca4f6a35743aa44fed91ba091b290ed0528"}
Sep 19 11:05:01.679796 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.679736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff" event={"ID":"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc","Type":"ContainerStarted","Data":"a8d2f1165234b4c9c0b0aaee89a1c745ba6f62b480234579ede1ed7b691163d0"}
Sep 19 11:05:01.681030 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.681005    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9" containerID="110733ff4bc116f8b94f6d7da7351d2a0ed382fb145605ee88656d3588d4ca93" exitCode=0
Sep 19 11:05:01.681148 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.681037    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"
Sep 19 11:05:01.681148 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.681069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4" event={"ID":"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9","Type":"ContainerDied","Data":"110733ff4bc116f8b94f6d7da7351d2a0ed382fb145605ee88656d3588d4ca93"}
Sep 19 11:05:01.681148 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.681104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4" event={"ID":"2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9","Type":"ContainerDied","Data":"3d9d38bef0705537fa64b40114ca5278ae7956f3aec89194c271a84d06ba2efa"}
Sep 19 11:05:01.681148 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.681125    2580 scope.go:117] "RemoveContainer" containerID="110733ff4bc116f8b94f6d7da7351d2a0ed382fb145605ee88656d3588d4ca93"
Sep 19 11:05:01.692493 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.692471    2580 scope.go:117] "RemoveContainer" containerID="110733ff4bc116f8b94f6d7da7351d2a0ed382fb145605ee88656d3588d4ca93"
Sep 19 11:05:01.692818 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:05:01.692792    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110733ff4bc116f8b94f6d7da7351d2a0ed382fb145605ee88656d3588d4ca93\": container with ID starting with 110733ff4bc116f8b94f6d7da7351d2a0ed382fb145605ee88656d3588d4ca93 not found: ID does not exist" containerID="110733ff4bc116f8b94f6d7da7351d2a0ed382fb145605ee88656d3588d4ca93"
Sep 19 11:05:01.692897 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.692830    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110733ff4bc116f8b94f6d7da7351d2a0ed382fb145605ee88656d3588d4ca93"} err="failed to get container status \"110733ff4bc116f8b94f6d7da7351d2a0ed382fb145605ee88656d3588d4ca93\": rpc error: code = NotFound desc = could not find container \"110733ff4bc116f8b94f6d7da7351d2a0ed382fb145605ee88656d3588d4ca93\": container with ID starting with 110733ff4bc116f8b94f6d7da7351d2a0ed382fb145605ee88656d3588d4ca93 not found: ID does not exist"
Sep 19 11:05:01.695276 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.695236    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff" podStartSLOduration=1.6952238419999999 podStartE2EDuration="1.695223842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:05:01.694182668 +0000 UTC m=+1746.352927866" watchObservedRunningTime="2026-09-19 11:05:01.695223842 +0000 UTC m=+1746.353969051"
Sep 19 11:05:01.705989 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.705961    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"]
Sep 19 11:05:01.710911 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.710889    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/sub-test-model-137f88-kserve-5b6f974545-rckn4"]
Sep 19 11:05:01.855700 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:01.855659    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9" path="/var/lib/kubelet/pods/2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9/volumes"
Sep 19 11:05:09.807687 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:09.807650    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:05:10.831154 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:10.831120    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:10.831557 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:10.831172    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:10.843831 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:10.843804    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:11.741625 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:11.741592    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:05:26.220914 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.220880    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"]
Sep 19 11:05:26.221292 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.221228    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9" containerName="main"
Sep 19 11:05:26.221292 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.221239    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9" containerName="main"
Sep 19 11:05:26.221365 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.221318    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c7452b1-28f3-4ffb-8d87-6b3b2d06efa9" containerName="main"
Sep 19 11:05:26.225710 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.225694    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.227865 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.227839    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9b3bef\"/\"openshift-service-ca.crt\""
Sep 19 11:05:26.227996 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.227979    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9b3bef\"/\"kube-root-ca.crt\""
Sep 19 11:05:26.228807 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.228788    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9b3bef\"/\"default-dockercfg-z6csr\""
Sep 19 11:05:26.228807 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.228802    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9b3bef\"/\"rate-test-model-9b3bef-kserve-self-signed-certs\""
Sep 19 11:05:26.232476 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.232453    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"]
Sep 19 11:05:26.296068 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.296038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92492c82-e2ee-49f2-a1bb-829aa40e0096-tls-certs\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.296259 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.296084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6b7\" (UniqueName: \"kubernetes.io/projected/92492c82-e2ee-49f2-a1bb-829aa40e0096-kube-api-access-hh6b7\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.296259 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.296133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-dshm\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.296259 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.296153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-model-cache\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.296259 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.296176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-tmp-dir\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.296259 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.296200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-home\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.397772 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.397722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-tmp-dir\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.398197 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.398171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-tmp-dir\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.398348 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.398261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-home\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.398415 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.398357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92492c82-e2ee-49f2-a1bb-829aa40e0096-tls-certs\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.398415 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.398400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6b7\" (UniqueName: \"kubernetes.io/projected/92492c82-e2ee-49f2-a1bb-829aa40e0096-kube-api-access-hh6b7\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.398512 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.398435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-dshm\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.398512 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.398463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-model-cache\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.398815 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.398794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-model-cache\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.399164 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.399137    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-home\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.401177 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.401152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-dshm\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.401555 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.401534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92492c82-e2ee-49f2-a1bb-829aa40e0096-tls-certs\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.406166 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.406145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6b7\" (UniqueName: \"kubernetes.io/projected/92492c82-e2ee-49f2-a1bb-829aa40e0096-kube-api-access-hh6b7\") pod \"rate-test-model-9b3bef-kserve-784f8559f5-nj5fm\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") " pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.417619 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.417583    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:05:26.535637 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.535549    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:26.810018 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:26.809996    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"]
Sep 19 11:05:26.812307 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:05:26.812276    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92492c82_e2ee_49f2_a1bb_829aa40e0096.slice/crio-b0d126f4d802563c2466a9a66eb622837eaf5ae16c009241070085550e00e48c WatchSource:0}: Error finding container b0d126f4d802563c2466a9a66eb622837eaf5ae16c009241070085550e00e48c: Status 404 returned error can't find the container with id b0d126f4d802563c2466a9a66eb622837eaf5ae16c009241070085550e00e48c
Sep 19 11:05:27.819311 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:27.819276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm" event={"ID":"92492c82-e2ee-49f2-a1bb-829aa40e0096","Type":"ContainerStarted","Data":"31fb0217b18b76b8079b722cdaa0a849cc80b8ee48db7a07d79cd7eea23c73bc"}
Sep 19 11:05:27.819786 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:27.819322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm" event={"ID":"92492c82-e2ee-49f2-a1bb-829aa40e0096","Type":"ContainerStarted","Data":"b0d126f4d802563c2466a9a66eb622837eaf5ae16c009241070085550e00e48c"}
Sep 19 11:05:27.835679 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:27.835633    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm" podStartSLOduration=1.835616982 podStartE2EDuration="1.835616982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:05:27.83337504 +0000 UTC m=+1772.492120212" watchObservedRunningTime="2026-09-19 11:05:27.835616982 +0000 UTC m=+1772.494362142"
Sep 19 11:05:30.114498 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:30.114464    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:05:36.535902 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:36.535854    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:36.536388 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:36.535916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:36.548893 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:36.548861    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:36.873986 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:36.873899    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:05:45.609265 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:45.609231    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:05:55.935705 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:55.935680    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 11:05:55.936147 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:55.936129    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 11:05:56.110907 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:05:56.110869    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:06:00.891227 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:00.891192    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"]
Sep 19 11:06:00.907708 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:00.907677    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:01.804532 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:01.804500    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"]
Sep 19 11:06:01.827417 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:01.827385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:02.249551 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:02.249515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:02.557221 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:02.557142    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:02.837496 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:02.837418    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:04.832158 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:04.832124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:05.608432 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:05.608389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:05.711459 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:05.711409    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:05.808892 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:05.808856    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:06.534977 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:06.534939    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:07.379748 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:07.379710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:07.480863 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:07.480828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:07.584472 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:07.584436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:07.830683 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:07.830647    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:07.936393 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:07.936353    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:11.412781 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:11.412734    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:06:15.911066 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:15.911011    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff" podUID="ca0f0c7e-6362-4223-a1dc-5e82108e4dbc" containerName="main" containerID="cri-o://add2e60304344cf5d5ab206bd6b65ca4f6a35743aa44fed91ba091b290ed0528" gracePeriod=45
Sep 19 11:06:16.072549 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.072512    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca0f0c7e-6362-4223-a1dc-5e82108e4dbc" containerID="add2e60304344cf5d5ab206bd6b65ca4f6a35743aa44fed91ba091b290ed0528" exitCode=0
Sep 19 11:06:16.072737 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.072599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff" event={"ID":"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc","Type":"ContainerDied","Data":"add2e60304344cf5d5ab206bd6b65ca4f6a35743aa44fed91ba091b290ed0528"}
Sep 19 11:06:16.152435 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.152412    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:06:16.244737 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.244696    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-dshm\") pod \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") "
Sep 19 11:06:16.244968 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.244786    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-home\") pod \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") "
Sep 19 11:06:16.244968 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.244815    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-626j8\" (UniqueName: \"kubernetes.io/projected/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-kube-api-access-626j8\") pod \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") "
Sep 19 11:06:16.244968 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.244833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-tmp-dir\") pod \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") "
Sep 19 11:06:16.244968 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.244876    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-model-cache\") pod \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") "
Sep 19 11:06:16.244968 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.244915    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-tls-certs\") pod \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\" (UID: \"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc\") "
Sep 19 11:06:16.245230 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.245058    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-home" (OuterVolumeSpecName: "home") pod "ca0f0c7e-6362-4223-a1dc-5e82108e4dbc" (UID: "ca0f0c7e-6362-4223-a1dc-5e82108e4dbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:06:16.245230 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.245086    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca0f0c7e-6362-4223-a1dc-5e82108e4dbc" (UID: "ca0f0c7e-6362-4223-a1dc-5e82108e4dbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:06:16.245230 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.245151    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:06:16.245230 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.245170    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:06:16.245230 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.245166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca0f0c7e-6362-4223-a1dc-5e82108e4dbc" (UID: "ca0f0c7e-6362-4223-a1dc-5e82108e4dbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:06:16.247136 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.247110    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca0f0c7e-6362-4223-a1dc-5e82108e4dbc" (UID: "ca0f0c7e-6362-4223-a1dc-5e82108e4dbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:06:16.247136 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.247129    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-kube-api-access-626j8" (OuterVolumeSpecName: "kube-api-access-626j8") pod "ca0f0c7e-6362-4223-a1dc-5e82108e4dbc" (UID: "ca0f0c7e-6362-4223-a1dc-5e82108e4dbc"). InnerVolumeSpecName "kube-api-access-626j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:06:16.247248 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.247128    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-dshm" (OuterVolumeSpecName: "dshm") pod "ca0f0c7e-6362-4223-a1dc-5e82108e4dbc" (UID: "ca0f0c7e-6362-4223-a1dc-5e82108e4dbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:06:16.346418 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.346384    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-626j8\" (UniqueName: \"kubernetes.io/projected/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-kube-api-access-626j8\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:06:16.346418 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.346412    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:06:16.346418 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.346428    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:06:16.346659 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.346438    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:06:16.824532 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.824476    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm" podUID="92492c82-e2ee-49f2-a1bb-829aa40e0096" containerName="main" containerID="cri-o://31fb0217b18b76b8079b722cdaa0a849cc80b8ee48db7a07d79cd7eea23c73bc" gracePeriod=45
Sep 19 11:06:16.863910 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:16.863874    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm" podUID="92492c82-e2ee-49f2-a1bb-829aa40e0096" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 19 11:06:17.074442 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.074416    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:06:17.078854 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.078787    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"
Sep 19 11:06:17.078854 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.078816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff" event={"ID":"ca0f0c7e-6362-4223-a1dc-5e82108e4dbc","Type":"ContainerDied","Data":"a8d2f1165234b4c9c0b0aaee89a1c745ba6f62b480234579ede1ed7b691163d0"}
Sep 19 11:06:17.079029 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.078877    2580 scope.go:117] "RemoveContainer" containerID="add2e60304344cf5d5ab206bd6b65ca4f6a35743aa44fed91ba091b290ed0528"
Sep 19 11:06:17.080331 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.080305    2580 generic.go:356] "Generic (PLEG): container finished" podID="92492c82-e2ee-49f2-a1bb-829aa40e0096" containerID="31fb0217b18b76b8079b722cdaa0a849cc80b8ee48db7a07d79cd7eea23c73bc" exitCode=0
Sep 19 11:06:17.080533 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.080352    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"
Sep 19 11:06:17.080533 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.080395    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm" event={"ID":"92492c82-e2ee-49f2-a1bb-829aa40e0096","Type":"ContainerDied","Data":"31fb0217b18b76b8079b722cdaa0a849cc80b8ee48db7a07d79cd7eea23c73bc"}
Sep 19 11:06:17.080533 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.080429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm" event={"ID":"92492c82-e2ee-49f2-a1bb-829aa40e0096","Type":"ContainerDied","Data":"b0d126f4d802563c2466a9a66eb622837eaf5ae16c009241070085550e00e48c"}
Sep 19 11:06:17.088288 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.088270    2580 scope.go:117] "RemoveContainer" containerID="31fb0217b18b76b8079b722cdaa0a849cc80b8ee48db7a07d79cd7eea23c73bc"
Sep 19 11:06:17.096999 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.096978    2580 scope.go:117] "RemoveContainer" containerID="31fb0217b18b76b8079b722cdaa0a849cc80b8ee48db7a07d79cd7eea23c73bc"
Sep 19 11:06:17.097330 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:06:17.097304    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31fb0217b18b76b8079b722cdaa0a849cc80b8ee48db7a07d79cd7eea23c73bc\": container with ID starting with 31fb0217b18b76b8079b722cdaa0a849cc80b8ee48db7a07d79cd7eea23c73bc not found: ID does not exist" containerID="31fb0217b18b76b8079b722cdaa0a849cc80b8ee48db7a07d79cd7eea23c73bc"
Sep 19 11:06:17.097414 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.097344    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31fb0217b18b76b8079b722cdaa0a849cc80b8ee48db7a07d79cd7eea23c73bc"} err="failed to get container status \"31fb0217b18b76b8079b722cdaa0a849cc80b8ee48db7a07d79cd7eea23c73bc\": rpc error: code = NotFound desc = could not find container \"31fb0217b18b76b8079b722cdaa0a849cc80b8ee48db7a07d79cd7eea23c73bc\": container with ID starting with 31fb0217b18b76b8079b722cdaa0a849cc80b8ee48db7a07d79cd7eea23c73bc not found: ID does not exist"
Sep 19 11:06:17.104811 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.104786    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"]
Sep 19 11:06:17.110635 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.110609    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-137f88/rate-test-model-137f88-kserve-779c974dc9-l76ff"]
Sep 19 11:06:17.152286 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.152250    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92492c82-e2ee-49f2-a1bb-829aa40e0096-tls-certs\") pod \"92492c82-e2ee-49f2-a1bb-829aa40e0096\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") "
Sep 19 11:06:17.152286 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.152286    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-home\") pod \"92492c82-e2ee-49f2-a1bb-829aa40e0096\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") "
Sep 19 11:06:17.152515 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.152334    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-model-cache\") pod \"92492c82-e2ee-49f2-a1bb-829aa40e0096\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") "
Sep 19 11:06:17.152515 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.152375    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-tmp-dir\") pod \"92492c82-e2ee-49f2-a1bb-829aa40e0096\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") "
Sep 19 11:06:17.152515 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.152411    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh6b7\" (UniqueName: \"kubernetes.io/projected/92492c82-e2ee-49f2-a1bb-829aa40e0096-kube-api-access-hh6b7\") pod \"92492c82-e2ee-49f2-a1bb-829aa40e0096\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") "
Sep 19 11:06:17.152515 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.152440    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-dshm\") pod \"92492c82-e2ee-49f2-a1bb-829aa40e0096\" (UID: \"92492c82-e2ee-49f2-a1bb-829aa40e0096\") "
Sep 19 11:06:17.152728 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.152664    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-home" (OuterVolumeSpecName: "home") pod "92492c82-e2ee-49f2-a1bb-829aa40e0096" (UID: "92492c82-e2ee-49f2-a1bb-829aa40e0096"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:06:17.152728 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.152670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-model-cache" (OuterVolumeSpecName: "model-cache") pod "92492c82-e2ee-49f2-a1bb-829aa40e0096" (UID: "92492c82-e2ee-49f2-a1bb-829aa40e0096"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:06:17.152728 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.152689    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92492c82-e2ee-49f2-a1bb-829aa40e0096" (UID: "92492c82-e2ee-49f2-a1bb-829aa40e0096"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:06:17.154818 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.154794    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-dshm" (OuterVolumeSpecName: "dshm") pod "92492c82-e2ee-49f2-a1bb-829aa40e0096" (UID: "92492c82-e2ee-49f2-a1bb-829aa40e0096"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:06:17.154818 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.154801    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92492c82-e2ee-49f2-a1bb-829aa40e0096-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92492c82-e2ee-49f2-a1bb-829aa40e0096" (UID: "92492c82-e2ee-49f2-a1bb-829aa40e0096"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:06:17.154947 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.154818    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92492c82-e2ee-49f2-a1bb-829aa40e0096-kube-api-access-hh6b7" (OuterVolumeSpecName: "kube-api-access-hh6b7") pod "92492c82-e2ee-49f2-a1bb-829aa40e0096" (UID: "92492c82-e2ee-49f2-a1bb-829aa40e0096"). InnerVolumeSpecName "kube-api-access-hh6b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:06:17.253300 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.253262    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92492c82-e2ee-49f2-a1bb-829aa40e0096-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:06:17.253300 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.253293    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:06:17.253300 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.253305    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:06:17.253548 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.253315    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:06:17.253548 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.253327    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh6b7\" (UniqueName: \"kubernetes.io/projected/92492c82-e2ee-49f2-a1bb-829aa40e0096-kube-api-access-hh6b7\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:06:17.253548 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.253338    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92492c82-e2ee-49f2-a1bb-829aa40e0096-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:06:17.400380 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.400350    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"]
Sep 19 11:06:17.402351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.402328    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9b3bef/rate-test-model-9b3bef-kserve-784f8559f5-nj5fm"]
Sep 19 11:06:17.861194 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.861151    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92492c82-e2ee-49f2-a1bb-829aa40e0096" path="/var/lib/kubelet/pods/92492c82-e2ee-49f2-a1bb-829aa40e0096/volumes"
Sep 19 11:06:17.861590 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:17.861570    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca0f0c7e-6362-4223-a1dc-5e82108e4dbc" path="/var/lib/kubelet/pods/ca0f0c7e-6362-4223-a1dc-5e82108e4dbc/volumes"
Sep 19 11:06:28.196811 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:28.196777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:28.811084 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:28.811046    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:29.988729 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:29.988685    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:30.709781 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:30.709732    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:30.827376 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:30.827337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:31.389053 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.389013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:31.489067 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.489026    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:31.673700 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.673604    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"]
Sep 19 11:06:31.674463 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.674417    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca0f0c7e-6362-4223-a1dc-5e82108e4dbc" containerName="main"
Sep 19 11:06:31.674463 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.674453    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca0f0c7e-6362-4223-a1dc-5e82108e4dbc" containerName="main"
Sep 19 11:06:31.674625 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.674474    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92492c82-e2ee-49f2-a1bb-829aa40e0096" containerName="main"
Sep 19 11:06:31.674625 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.674483    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="92492c82-e2ee-49f2-a1bb-829aa40e0096" containerName="main"
Sep 19 11:06:31.674625 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.674574    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca0f0c7e-6362-4223-a1dc-5e82108e4dbc" containerName="main"
Sep 19 11:06:31.674625 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.674586    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="92492c82-e2ee-49f2-a1bb-829aa40e0096" containerName="main"
Sep 19 11:06:31.678018 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.677996    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.680092 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.680068    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-063e1d-openshift-default-dockercfg-g979v\""
Sep 19 11:06:31.680823 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.680733    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 19 11:06:31.687983 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.687958    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"]
Sep 19 11:06:31.702385 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.702353    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:31.784499 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.784466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-data\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.784688 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.784505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-workload-certs\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.784688 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.784532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-credential-socket\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.784688 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.784550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fe6072-044c-490d-ad9c-30ef7d235b55-istiod-ca-cert\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.784688 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.784602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-envoy\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.784688 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.784619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-token\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.784923 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.784695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpcgz\" (UniqueName: \"kubernetes.io/projected/41fe6072-044c-490d-ad9c-30ef7d235b55-kube-api-access-lpcgz\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.784923 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.784743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-workload-socket\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.784923 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.784791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-podinfo\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.886127 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.886094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-workload-certs\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.886328 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.886133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-credential-socket\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.886328 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.886151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fe6072-044c-490d-ad9c-30ef7d235b55-istiod-ca-cert\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.886328 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.886172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-envoy\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.886328 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.886188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-token\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.886328 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.886238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpcgz\" (UniqueName: \"kubernetes.io/projected/41fe6072-044c-490d-ad9c-30ef7d235b55-kube-api-access-lpcgz\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.886328 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.886289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-workload-socket\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.886328 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.886312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-podinfo\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.886702 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.886482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-data\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.886702 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.886624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-credential-socket\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.886702 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.886643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-workload-certs\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.886840 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.886727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-workload-socket\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.886882 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.886859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-data\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.887249 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.887228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fe6072-044c-490d-ad9c-30ef7d235b55-istiod-ca-cert\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.888855 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.888840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-envoy\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.888912 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.888888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-podinfo\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.893618 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.893588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-token\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.893745 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.893640    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpcgz\" (UniqueName: \"kubernetes.io/projected/41fe6072-044c-490d-ad9c-30ef7d235b55-kube-api-access-lpcgz\") pod \"e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") " pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:31.989921 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:31.989871    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:32.202084 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:32.202049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:32.269123 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:32.269093    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"]
Sep 19 11:06:32.275306 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:32.275265    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 11:06:32.275433 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:32.275341    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 11:06:32.275433 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:32.275377    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 11:06:32.335446 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:32.335400    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:32.699804 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:32.699742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:33.166676 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:33.166574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" event={"ID":"41fe6072-044c-490d-ad9c-30ef7d235b55","Type":"ContainerStarted","Data":"faa42f013c3e431ef115c81b2e21d3c697081e146402b558dbd79c0deee2b815"}
Sep 19 11:06:33.166676 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:33.166643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" event={"ID":"41fe6072-044c-490d-ad9c-30ef7d235b55","Type":"ContainerStarted","Data":"35f5933f93fef0704f547eae0955a3fcf4ff2349fefc6de081cd6da77de12455"}
Sep 19 11:06:33.185028 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:33.184975    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" podStartSLOduration=2.1849595 podStartE2EDuration="2.1849595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:06:33.183492289 +0000 UTC m=+1837.842237450" watchObservedRunningTime="2026-09-19 11:06:33.1849595 +0000 UTC m=+1837.843704660"
Sep 19 11:06:33.991095 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:33.991056    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:33.996356 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:33.996327    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:34.172297 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:34.172254    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:34.173269 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:34.173249    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:34.318607 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:34.318515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:35.020422 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:35.020383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:35.119020 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:35.118983    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:35.219416 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:35.219384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:35.765835 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:35.765796    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:36.721927 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:36.721893    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:37.344964 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.344930    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"]
Sep 19 11:06:37.347836 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.347818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:37.350038 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.350014    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-063e1d-serving-cert\""
Sep 19 11:06:37.350861 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.350840    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g2pqv\""
Sep 19 11:06:37.350984 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.350880    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 19 11:06:37.350984 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.350879    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 11:06:37.350984 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.350840    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 19 11:06:37.357518 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.357491    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"]
Sep 19 11:06:37.391333 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.391302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:37.440364 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.440326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pggrm\" (UniqueName: \"kubernetes.io/projected/f168f4fb-925d-467f-80aa-27c3e74a7093-kube-api-access-pggrm\") pod \"maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:37.440532 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.440373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f168f4fb-925d-467f-80aa-27c3e74a7093-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:37.440532 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.440394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-metrics-certs\") pod \"maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:37.440640 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.440555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-maas-api-tls\") pod \"maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:37.492811 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.492737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:37.541236 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.541205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f168f4fb-925d-467f-80aa-27c3e74a7093-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:37.541406 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.541240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-metrics-certs\") pod \"maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:37.541406 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.541312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-maas-api-tls\") pod \"maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:37.541406 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.541329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pggrm\" (UniqueName: \"kubernetes.io/projected/f168f4fb-925d-467f-80aa-27c3e74a7093-kube-api-access-pggrm\") pod \"maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:37.541531 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:06:37.541438    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-serving-cert: secret "maas-api-e2e-ipp-b-063e1d-serving-cert" not found
Sep 19 11:06:37.541531 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:06:37.541515    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-maas-api-tls podName:f168f4fb-925d-467f-80aa-27c3e74a7093 nodeName:}" failed. No retries permitted until 2026-09-19 11:06:38.041494259 +0000 UTC m=+1842.700239400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-maas-api-tls") pod "maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2" (UID: "f168f4fb-925d-467f-80aa-27c3e74a7093") : secret "maas-api-e2e-ipp-b-063e1d-serving-cert" not found
Sep 19 11:06:37.541943 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.541919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f168f4fb-925d-467f-80aa-27c3e74a7093-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:37.544044 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.544019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-metrics-certs\") pod \"maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:37.549314 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.549289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pggrm\" (UniqueName: \"kubernetes.io/projected/f168f4fb-925d-467f-80aa-27c3e74a7093-kube-api-access-pggrm\") pod \"maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:37.831419 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.831373    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:37.930045 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.930002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:37.933226 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.933193    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"]
Sep 19 11:06:37.936710 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.936691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"
Sep 19 11:06:37.938846 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.938821    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-063e1d-dockercfg-bwb89\""
Sep 19 11:06:37.938976 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.938821    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-063e1d\""
Sep 19 11:06:37.945279 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.945250    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"]
Sep 19 11:06:37.973071 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.973033    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"]
Sep 19 11:06:37.976578 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.976558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"
Sep 19 11:06:37.986832 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:37.986800    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"]
Sep 19 11:06:38.003980 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.003943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:06:38.045695 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.045663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchmh\" (UniqueName: \"kubernetes.io/projected/440f5528-193d-4066-b3d4-a57e0464ce83-kube-api-access-fchmh\") pod \"payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl\" (UID: \"440f5528-193d-4066-b3d4-a57e0464ce83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"
Sep 19 11:06:38.045863 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.045712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440f5528-193d-4066-b3d4-a57e0464ce83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl\" (UID: \"440f5528-193d-4066-b3d4-a57e0464ce83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"
Sep 19 11:06:38.045863 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.045820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-maas-api-tls\") pod \"maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:38.048408 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.048385    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-maas-api-tls\") pod \"maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:38.147309 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.147204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchmh\" (UniqueName: \"kubernetes.io/projected/440f5528-193d-4066-b3d4-a57e0464ce83-kube-api-access-fchmh\") pod \"payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl\" (UID: \"440f5528-193d-4066-b3d4-a57e0464ce83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"
Sep 19 11:06:38.147309 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.147254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d1f3b1-5768-49e2-abae-5231a55fcf85-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz\" (UID: \"25d1f3b1-5768-49e2-abae-5231a55fcf85\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"
Sep 19 11:06:38.147309 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.147287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440f5528-193d-4066-b3d4-a57e0464ce83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl\" (UID: \"440f5528-193d-4066-b3d4-a57e0464ce83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"
Sep 19 11:06:38.147615 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.147324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8pzx\" (UniqueName: \"kubernetes.io/projected/25d1f3b1-5768-49e2-abae-5231a55fcf85-kube-api-access-w8pzx\") pod \"payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz\" (UID: \"25d1f3b1-5768-49e2-abae-5231a55fcf85\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"
Sep 19 11:06:38.148038 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.148018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440f5528-193d-4066-b3d4-a57e0464ce83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl\" (UID: \"440f5528-193d-4066-b3d4-a57e0464ce83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"
Sep 19 11:06:38.155414 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.155390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchmh\" (UniqueName: \"kubernetes.io/projected/440f5528-193d-4066-b3d4-a57e0464ce83-kube-api-access-fchmh\") pod \"payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl\" (UID: \"440f5528-193d-4066-b3d4-a57e0464ce83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"
Sep 19 11:06:38.248022 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.247985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8pzx\" (UniqueName: \"kubernetes.io/projected/25d1f3b1-5768-49e2-abae-5231a55fcf85-kube-api-access-w8pzx\") pod \"payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz\" (UID: \"25d1f3b1-5768-49e2-abae-5231a55fcf85\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"
Sep 19 11:06:38.248201 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.248069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d1f3b1-5768-49e2-abae-5231a55fcf85-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz\" (UID: \"25d1f3b1-5768-49e2-abae-5231a55fcf85\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"
Sep 19 11:06:38.248769 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.248735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d1f3b1-5768-49e2-abae-5231a55fcf85-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz\" (UID: \"25d1f3b1-5768-49e2-abae-5231a55fcf85\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"
Sep 19 11:06:38.248834 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.248740    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"
Sep 19 11:06:38.255695 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.255667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8pzx\" (UniqueName: \"kubernetes.io/projected/25d1f3b1-5768-49e2-abae-5231a55fcf85-kube-api-access-w8pzx\") pod \"payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz\" (UID: \"25d1f3b1-5768-49e2-abae-5231a55fcf85\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"
Sep 19 11:06:38.261484 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.261452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:38.288490 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.288446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"
Sep 19 11:06:38.529521 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.529468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"]
Sep 19 11:06:38.531984 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:06:38.531955    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440f5528_193d_4066_b3d4_a57e0464ce83.slice/crio-2f97a3088803a0a5d3942b81c7d1d56eb47d8cfdb2c173b3e231ca77f9d2dc1e WatchSource:0}: Error finding container 2f97a3088803a0a5d3942b81c7d1d56eb47d8cfdb2c173b3e231ca77f9d2dc1e: Status 404 returned error can't find the container with id 2f97a3088803a0a5d3942b81c7d1d56eb47d8cfdb2c173b3e231ca77f9d2dc1e
Sep 19 11:06:38.550802 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.550768    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"]
Sep 19 11:06:38.552159 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:06:38.552132    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf168f4fb_925d_467f_80aa_27c3e74a7093.slice/crio-7df1c9cc1badded696eae2e3f2f2247322d8429cb996ba28fad0d591f6020c35 WatchSource:0}: Error finding container 7df1c9cc1badded696eae2e3f2f2247322d8429cb996ba28fad0d591f6020c35: Status 404 returned error can't find the container with id 7df1c9cc1badded696eae2e3f2f2247322d8429cb996ba28fad0d591f6020c35
Sep 19 11:06:38.553652 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.553632    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 11:06:38.576862 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:38.576834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"]
Sep 19 11:06:38.580229 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:06:38.580199    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d1f3b1_5768_49e2_abae_5231a55fcf85.slice/crio-1d419ac581a576782805e0781736658c504dc29aa31108e67c86369dd1f4dc5c WatchSource:0}: Error finding container 1d419ac581a576782805e0781736658c504dc29aa31108e67c86369dd1f4dc5c: Status 404 returned error can't find the container with id 1d419ac581a576782805e0781736658c504dc29aa31108e67c86369dd1f4dc5c
Sep 19 11:06:39.200323 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:39.200286    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz" event={"ID":"25d1f3b1-5768-49e2-abae-5231a55fcf85","Type":"ContainerStarted","Data":"3b257cfb317ae54fbeb160a40a51df0e84e558ab98ed41040319c0c98e683392"}
Sep 19 11:06:39.200844 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:39.200350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz" event={"ID":"25d1f3b1-5768-49e2-abae-5231a55fcf85","Type":"ContainerStarted","Data":"1d419ac581a576782805e0781736658c504dc29aa31108e67c86369dd1f4dc5c"}
Sep 19 11:06:39.200844 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:39.200379    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"
Sep 19 11:06:39.202014 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:39.201985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2" event={"ID":"f168f4fb-925d-467f-80aa-27c3e74a7093","Type":"ContainerStarted","Data":"698a91bfbb3ae66cb81b5ea941b81dd026ebd06ab0dedd1ba5fb237ab522d5f9"}
Sep 19 11:06:39.202145 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:39.202036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2" event={"ID":"f168f4fb-925d-467f-80aa-27c3e74a7093","Type":"ContainerStarted","Data":"7df1c9cc1badded696eae2e3f2f2247322d8429cb996ba28fad0d591f6020c35"}
Sep 19 11:06:39.202145 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:39.202078    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:39.203612 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:39.203590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl" event={"ID":"440f5528-193d-4066-b3d4-a57e0464ce83","Type":"ContainerStarted","Data":"db342385c3f231b3f67f7cd5f256eb3b46a9076fc38ba36f9583880128a431f3"}
Sep 19 11:06:39.203711 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:39.203634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl" event={"ID":"440f5528-193d-4066-b3d4-a57e0464ce83","Type":"ContainerStarted","Data":"2f97a3088803a0a5d3942b81c7d1d56eb47d8cfdb2c173b3e231ca77f9d2dc1e"}
Sep 19 11:06:39.203711 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:39.203667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"
Sep 19 11:06:39.217202 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:39.217157    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz" podStartSLOduration=2.2171449069999998 podStartE2EDuration="2.217144907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:06:39.215800584 +0000 UTC m=+1843.874545744" watchObservedRunningTime="2026-09-19 11:06:39.217144907 +0000 UTC m=+1843.875890066"
Sep 19 11:06:39.231331 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:39.231281    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2" podStartSLOduration=1.913298983 podStartE2EDuration="2.231268002s" totalImagesPullingTime="317.969019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:06:39.230510262 +0000 UTC m=+1843.889255436" watchObservedRunningTime="2026-09-19 11:06:39.231268002 +0000 UTC m=+1843.890013161"
Sep 19 11:06:39.244839 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:39.244785    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl" podStartSLOduration=2.244771071 podStartE2EDuration="2.244771071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:06:39.243914121 +0000 UTC m=+1843.902659280" watchObservedRunningTime="2026-09-19 11:06:39.244771071 +0000 UTC m=+1843.903516223"
Sep 19 11:06:42.412366 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:42.412322    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:06:45.215842 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:45.215807    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:06:50.212471 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:50.212438    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"
Sep 19 11:06:50.212963 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:50.212632    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"
Sep 19 11:06:50.213893 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:50.213870    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 19 11:06:50.214032 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:50.213923    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 19 11:06:52.276460 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:52.276366    2580 generic.go:356] "Generic (PLEG): container finished" podID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerID="faa42f013c3e431ef115c81b2e21d3c697081e146402b558dbd79c0deee2b815" exitCode=0
Sep 19 11:06:52.276460 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:52.276444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" event={"ID":"41fe6072-044c-490d-ad9c-30ef7d235b55","Type":"ContainerDied","Data":"faa42f013c3e431ef115c81b2e21d3c697081e146402b558dbd79c0deee2b815"}
Sep 19 11:06:52.276930 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:52.276835    2580 scope.go:117] "RemoveContainer" containerID="faa42f013c3e431ef115c81b2e21d3c697081e146402b558dbd79c0deee2b815"
Sep 19 11:06:52.277907 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:52.277864    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 11:06:52.277984 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:52.277948    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 11:06:52.278029 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:52.277983    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 19 11:06:53.287245 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:53.287203    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" event={"ID":"41fe6072-044c-490d-ad9c-30ef7d235b55","Type":"ContainerStarted","Data":"c15c72d0062371eaba326b35f8a8c34ea01892c5140ef1cd0ecc08f296969310"}
Sep 19 11:06:53.990672 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:53.990626    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:53.990932 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:53.990912    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 19 11:06:53.990991 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:53.990974    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 19 11:06:54.991146 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:54.991101    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 19 11:06:54.991559 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:54.991200    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 19 11:06:55.990795 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:55.990740    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 19 11:06:55.990988 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:55.990823    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 19 11:06:56.138077 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:56.138042    2580 scope.go:117] "RemoveContainer" containerID="e6337e91b01ac2789ad7bf200fe45453524557da2a13d7b6667597ef033ad38c"
Sep 19 11:06:56.990470 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:56.990411    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 19 11:06:56.990664 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:56.990492    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 19 11:06:57.990572 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:57.990530    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 19 11:06:57.991021 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:57.990595    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 19 11:06:58.994880 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:58.994846    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:58.995320 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:58.995127    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:58.995729 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:58.995712    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:06:59.270858 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:06:59.270746    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:00.246603 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:00.246564    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:00.617174 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:00.617068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:01.342448 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:01.342416    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:02.125323 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:02.125285    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:05.229254 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:05.229221    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz_25d1f3b1-5768-49e2-abae-5231a55fcf85/payload-processing/0.log"
Sep 19 11:07:08.212412 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.212379    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"]
Sep 19 11:07:08.216001 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.215983    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.219156 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.219128    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b52037\"/\"openshift-service-ca.crt\""
Sep 19 11:07:08.219324 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.219182    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b52037\"/\"ipp-route-b52037-kserve-self-signed-certs\""
Sep 19 11:07:08.219324 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.219200    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b52037\"/\"default-dockercfg-bf56j\""
Sep 19 11:07:08.219324 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.219216    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b52037\"/\"kube-root-ca.crt\""
Sep 19 11:07:08.223896 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.223869    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"]
Sep 19 11:07:08.237279 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.237246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:08.316076 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.316041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f9153f-a907-4e31-a1f9-77766b5b6ce7-tls-certs\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.316269 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.316083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-home\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.316269 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.316145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-dshm\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.316269 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.316187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-tmp-dir\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.316269 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.316211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-model-cache\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.316269 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.316245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcg5k\" (UniqueName: \"kubernetes.io/projected/48f9153f-a907-4e31-a1f9-77766b5b6ce7-kube-api-access-wcg5k\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.417555 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.417509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcg5k\" (UniqueName: \"kubernetes.io/projected/48f9153f-a907-4e31-a1f9-77766b5b6ce7-kube-api-access-wcg5k\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.417746 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.417579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f9153f-a907-4e31-a1f9-77766b5b6ce7-tls-certs\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.417746 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.417621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-home\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.417930 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.417808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-dshm\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.417930 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.417865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-tmp-dir\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.417930 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.417901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-model-cache\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.418128 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.418104    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-home\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.418198 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.418179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-tmp-dir\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.418261 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.418246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-model-cache\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.420214 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.420195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-dshm\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.420493 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.420479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f9153f-a907-4e31-a1f9-77766b5b6ce7-tls-certs\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.424691 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.424665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcg5k\" (UniqueName: \"kubernetes.io/projected/48f9153f-a907-4e31-a1f9-77766b5b6ce7-kube-api-access-wcg5k\") pod \"ipp-route-b52037-kserve-668964d5f5-mtxng\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") " pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.528526 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.528428    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:08.802375 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:08.801728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"]
Sep 19 11:07:08.804503 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:07:08.804471    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48f9153f_a907_4e31_a1f9_77766b5b6ce7.slice/crio-65a0fd5668539fab2cd04b920ad9449a400db0e8561b72e72895d460ea816736 WatchSource:0}: Error finding container 65a0fd5668539fab2cd04b920ad9449a400db0e8561b72e72895d460ea816736: Status 404 returned error can't find the container with id 65a0fd5668539fab2cd04b920ad9449a400db0e8561b72e72895d460ea816736
Sep 19 11:07:09.369420 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:09.369380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng" event={"ID":"48f9153f-a907-4e31-a1f9-77766b5b6ce7","Type":"ContainerStarted","Data":"a50e5c2c00ee3321083e10a38e2aebfb95da3ca246691733da1c79e9a6e9f7e0"}
Sep 19 11:07:09.369861 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:09.369428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng" event={"ID":"48f9153f-a907-4e31-a1f9-77766b5b6ce7","Type":"ContainerStarted","Data":"65a0fd5668539fab2cd04b920ad9449a400db0e8561b72e72895d460ea816736"}
Sep 19 11:07:09.385084 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:09.385025    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng" podStartSLOduration=1.385008448 podStartE2EDuration="1.385008448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:07:09.384127771 +0000 UTC m=+1874.042872932" watchObservedRunningTime="2026-09-19 11:07:09.385008448 +0000 UTC m=+1874.043753607"
Sep 19 11:07:12.216387 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:12.216348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:07:18.529590 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:18.529486    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:18.529590 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:18.529547    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:18.542824 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:18.542795    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:19.438465 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:19.438426    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:07:25.309881 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:25.309844    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:07:31.175707 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:31.175671    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz_25d1f3b1-5768-49e2-abae-5231a55fcf85/payload-processing/0.log"
Sep 19 11:07:31.440962 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:31.440925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:31.741075 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:31.740981    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:32.070042 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:32.069959    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:33.734130 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:33.734094    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:34.505384 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:34.505349    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:34.604730 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:34.604691    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:35.264100 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:35.264062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:35.986936 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:35.986902    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:36.722065 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:36.722027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:37.333613 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:37.333576    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:37.385463 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:37.385422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:37.772936 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:37.772901    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:37.873493 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:37.873461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:38.229617 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:38.229586    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:43.423110 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:43.423071    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:07:56.118289 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:56.118255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:56.172286 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:56.172243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:59.309855 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:59.309808    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:07:59.438260 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:07:59.438218    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:00.231546 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:00.231509    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:00.934305 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:00.934267    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:01.712145 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:01.712105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:02.419591 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:02.419555    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:03.045261 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:03.045228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:03.823265 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:03.823229    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:04.535435 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:04.535397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:06.015079 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:06.015049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:06.588704 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:06.588669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:06.664776 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:06.664715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:06.726963 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:06.726921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:06.783742 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:06.783708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:07.140117 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:07.140082    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:07.240439 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:07.240405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:07.914942 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:07.914907    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:08.552029 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:08.551991    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:09.418208 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:09.418167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:10.055863 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:10.055823    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:10.945979 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:10.945948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:10.999066 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:10.999036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:11.224734 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:11.224624    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:08:11.228270 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:11.224684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:12.700776 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:12.700717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:12.801431 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:12.801392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:12.975136 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:12.975046    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:16.731510 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:16.731469    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:17.527068 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:17.527033    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:18.234894 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:18.234860    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:18.925184 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:18.925144    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:19.615460 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:19.615418    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:20.331159 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:20.331118    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:29.708401 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:29.708360    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:08:32.085213 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:32.085179    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:32.157075 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:32.157023    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:32.212824 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:32.212776    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:32.268436 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:32.268400    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:32.639318 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:32.639283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:32.739282 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:32.739241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:33.312913 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:33.312872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:34.108388 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:34.108343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:34.719546 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:34.719499    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:37.621710 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:37.621673    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"]
Sep 19 11:08:37.667908 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:37.667872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:08:42.918555 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:42.918517    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:08:52.642191 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.642089    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng" podUID="48f9153f-a907-4e31-a1f9-77766b5b6ce7" containerName="main" containerID="cri-o://a50e5c2c00ee3321083e10a38e2aebfb95da3ca246691733da1c79e9a6e9f7e0" gracePeriod=45
Sep 19 11:08:52.893044 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.892976    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:08:52.903971 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.903942    2580 generic.go:356] "Generic (PLEG): container finished" podID="48f9153f-a907-4e31-a1f9-77766b5b6ce7" containerID="a50e5c2c00ee3321083e10a38e2aebfb95da3ca246691733da1c79e9a6e9f7e0" exitCode=0
Sep 19 11:08:52.904118 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.903983    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"
Sep 19 11:08:52.904118 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.904030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng" event={"ID":"48f9153f-a907-4e31-a1f9-77766b5b6ce7","Type":"ContainerDied","Data":"a50e5c2c00ee3321083e10a38e2aebfb95da3ca246691733da1c79e9a6e9f7e0"}
Sep 19 11:08:52.904118 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.904076    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng" event={"ID":"48f9153f-a907-4e31-a1f9-77766b5b6ce7","Type":"ContainerDied","Data":"65a0fd5668539fab2cd04b920ad9449a400db0e8561b72e72895d460ea816736"}
Sep 19 11:08:52.904118 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.904095    2580 scope.go:117] "RemoveContainer" containerID="a50e5c2c00ee3321083e10a38e2aebfb95da3ca246691733da1c79e9a6e9f7e0"
Sep 19 11:08:52.917790 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.917738    2580 scope.go:117] "RemoveContainer" containerID="a50e5c2c00ee3321083e10a38e2aebfb95da3ca246691733da1c79e9a6e9f7e0"
Sep 19 11:08:52.918203 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:08:52.918179    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50e5c2c00ee3321083e10a38e2aebfb95da3ca246691733da1c79e9a6e9f7e0\": container with ID starting with a50e5c2c00ee3321083e10a38e2aebfb95da3ca246691733da1c79e9a6e9f7e0 not found: ID does not exist" containerID="a50e5c2c00ee3321083e10a38e2aebfb95da3ca246691733da1c79e9a6e9f7e0"
Sep 19 11:08:52.918322 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.918212    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50e5c2c00ee3321083e10a38e2aebfb95da3ca246691733da1c79e9a6e9f7e0"} err="failed to get container status \"a50e5c2c00ee3321083e10a38e2aebfb95da3ca246691733da1c79e9a6e9f7e0\": rpc error: code = NotFound desc = could not find container \"a50e5c2c00ee3321083e10a38e2aebfb95da3ca246691733da1c79e9a6e9f7e0\": container with ID starting with a50e5c2c00ee3321083e10a38e2aebfb95da3ca246691733da1c79e9a6e9f7e0 not found: ID does not exist"
Sep 19 11:08:52.965462 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.965433    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-model-cache\") pod \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") "
Sep 19 11:08:52.965462 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.965466    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-home\") pod \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") "
Sep 19 11:08:52.965731 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.965502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-dshm\") pod \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") "
Sep 19 11:08:52.965731 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.965571    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f9153f-a907-4e31-a1f9-77766b5b6ce7-tls-certs\") pod \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") "
Sep 19 11:08:52.965731 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.965599    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcg5k\" (UniqueName: \"kubernetes.io/projected/48f9153f-a907-4e31-a1f9-77766b5b6ce7-kube-api-access-wcg5k\") pod \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") "
Sep 19 11:08:52.965731 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.965649    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-tmp-dir\") pod \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\" (UID: \"48f9153f-a907-4e31-a1f9-77766b5b6ce7\") "
Sep 19 11:08:52.965943 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.965822    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-home" (OuterVolumeSpecName: "home") pod "48f9153f-a907-4e31-a1f9-77766b5b6ce7" (UID: "48f9153f-a907-4e31-a1f9-77766b5b6ce7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:08:52.965943 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.965836    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-model-cache" (OuterVolumeSpecName: "model-cache") pod "48f9153f-a907-4e31-a1f9-77766b5b6ce7" (UID: "48f9153f-a907-4e31-a1f9-77766b5b6ce7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:08:52.966017 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.965964    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:08:52.966017 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.965982    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:08:52.966216 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.966194    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48f9153f-a907-4e31-a1f9-77766b5b6ce7" (UID: "48f9153f-a907-4e31-a1f9-77766b5b6ce7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:08:52.968024 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.967998    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-dshm" (OuterVolumeSpecName: "dshm") pod "48f9153f-a907-4e31-a1f9-77766b5b6ce7" (UID: "48f9153f-a907-4e31-a1f9-77766b5b6ce7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:08:52.968119 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.968023    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f9153f-a907-4e31-a1f9-77766b5b6ce7-kube-api-access-wcg5k" (OuterVolumeSpecName: "kube-api-access-wcg5k") pod "48f9153f-a907-4e31-a1f9-77766b5b6ce7" (UID: "48f9153f-a907-4e31-a1f9-77766b5b6ce7"). InnerVolumeSpecName "kube-api-access-wcg5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:08:52.968119 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:52.968078    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48f9153f-a907-4e31-a1f9-77766b5b6ce7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48f9153f-a907-4e31-a1f9-77766b5b6ce7" (UID: "48f9153f-a907-4e31-a1f9-77766b5b6ce7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:08:53.066995 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:53.066956    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:08:53.066995 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:53.066989    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f9153f-a907-4e31-a1f9-77766b5b6ce7-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:08:53.067205 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:53.067000    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcg5k\" (UniqueName: \"kubernetes.io/projected/48f9153f-a907-4e31-a1f9-77766b5b6ce7-kube-api-access-wcg5k\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:08:53.067205 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:53.067021    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f9153f-a907-4e31-a1f9-77766b5b6ce7-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:08:53.224006 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:53.223977    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"]
Sep 19 11:08:53.227372 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:53.227348    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b52037/ipp-route-b52037-kserve-668964d5f5-mtxng"]
Sep 19 11:08:53.856351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:08:53.856316    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48f9153f-a907-4e31-a1f9-77766b5b6ce7" path="/var/lib/kubelet/pods/48f9153f-a907-4e31-a1f9-77766b5b6ce7/volumes"
Sep 19 11:09:01.271396 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:01.271340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:01.373075 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:01.373043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:13.818439 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:13.818404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:14.673875 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:14.673828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:20.197368 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:20.197331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:25.621234 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:25.621195    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:25.677197 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:25.677158    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:25.885093 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:25.884997    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:26.433575 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:26.433536    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:27.220534 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:27.220495    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:27.827035 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:27.826992    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:28.532163 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:28.532116    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:29.195307 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:29.195262    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:36.750919 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:36.750880    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"]
Sep 19 11:09:36.751461 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:36.751136    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2" podUID="f168f4fb-925d-467f-80aa-27c3e74a7093" containerName="maas-api" containerID="cri-o://698a91bfbb3ae66cb81b5ea941b81dd026ebd06ab0dedd1ba5fb237ab522d5f9" gracePeriod=30
Sep 19 11:09:36.782512 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:36.782478    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:36.813220 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:36.812461    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"]
Sep 19 11:09:36.814049 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:36.814008    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz" podUID="25d1f3b1-5768-49e2-abae-5231a55fcf85" containerName="payload-processing" containerID="cri-o://3b257cfb317ae54fbeb160a40a51df0e84e558ab98ed41040319c0c98e683392" gracePeriod=30
Sep 19 11:09:36.819418 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:36.819392    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"]
Sep 19 11:09:36.819677 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:36.819629    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl" podUID="440f5528-193d-4066-b3d4-a57e0464ce83" containerName="payload-pre-processing" containerID="cri-o://db342385c3f231b3f67f7cd5f256eb3b46a9076fc38ba36f9583880128a431f3" gracePeriod=30
Sep 19 11:09:36.853286 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:36.853249    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:36.907532 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:36.907497    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:36.968975 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:36.968942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:37.024339 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.024307    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:37.101151 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.101122    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:09:37.115159 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.115135    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"
Sep 19 11:09:37.128981 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.128955    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"
Sep 19 11:09:37.133601 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.133562    2580 generic.go:356] "Generic (PLEG): container finished" podID="25d1f3b1-5768-49e2-abae-5231a55fcf85" containerID="3b257cfb317ae54fbeb160a40a51df0e84e558ab98ed41040319c0c98e683392" exitCode=0
Sep 19 11:09:37.133788 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.133644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz" event={"ID":"25d1f3b1-5768-49e2-abae-5231a55fcf85","Type":"ContainerDied","Data":"3b257cfb317ae54fbeb160a40a51df0e84e558ab98ed41040319c0c98e683392"}
Sep 19 11:09:37.133788 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.133720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz" event={"ID":"25d1f3b1-5768-49e2-abae-5231a55fcf85","Type":"ContainerDied","Data":"1d419ac581a576782805e0781736658c504dc29aa31108e67c86369dd1f4dc5c"}
Sep 19 11:09:37.133902 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.133792    2580 scope.go:117] "RemoveContainer" containerID="3b257cfb317ae54fbeb160a40a51df0e84e558ab98ed41040319c0c98e683392"
Sep 19 11:09:37.133951 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.133943    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"
Sep 19 11:09:37.136293 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.136250    2580 generic.go:356] "Generic (PLEG): container finished" podID="f168f4fb-925d-467f-80aa-27c3e74a7093" containerID="698a91bfbb3ae66cb81b5ea941b81dd026ebd06ab0dedd1ba5fb237ab522d5f9" exitCode=0
Sep 19 11:09:37.136594 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.136438    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"
Sep 19 11:09:37.136594 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.136484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2" event={"ID":"f168f4fb-925d-467f-80aa-27c3e74a7093","Type":"ContainerDied","Data":"698a91bfbb3ae66cb81b5ea941b81dd026ebd06ab0dedd1ba5fb237ab522d5f9"}
Sep 19 11:09:37.136594 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.136534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2" event={"ID":"f168f4fb-925d-467f-80aa-27c3e74a7093","Type":"ContainerDied","Data":"7df1c9cc1badded696eae2e3f2f2247322d8429cb996ba28fad0d591f6020c35"}
Sep 19 11:09:37.138783 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.138705    2580 generic.go:356] "Generic (PLEG): container finished" podID="440f5528-193d-4066-b3d4-a57e0464ce83" containerID="db342385c3f231b3f67f7cd5f256eb3b46a9076fc38ba36f9583880128a431f3" exitCode=0
Sep 19 11:09:37.138783 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.138729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl" event={"ID":"440f5528-193d-4066-b3d4-a57e0464ce83","Type":"ContainerDied","Data":"db342385c3f231b3f67f7cd5f256eb3b46a9076fc38ba36f9583880128a431f3"}
Sep 19 11:09:37.138783 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.138773    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"
Sep 19 11:09:37.138973 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.138798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl" event={"ID":"440f5528-193d-4066-b3d4-a57e0464ce83","Type":"ContainerDied","Data":"2f97a3088803a0a5d3942b81c7d1d56eb47d8cfdb2c173b3e231ca77f9d2dc1e"}
Sep 19 11:09:37.145172 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.145153    2580 scope.go:117] "RemoveContainer" containerID="3b257cfb317ae54fbeb160a40a51df0e84e558ab98ed41040319c0c98e683392"
Sep 19 11:09:37.145481 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:09:37.145462    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b257cfb317ae54fbeb160a40a51df0e84e558ab98ed41040319c0c98e683392\": container with ID starting with 3b257cfb317ae54fbeb160a40a51df0e84e558ab98ed41040319c0c98e683392 not found: ID does not exist" containerID="3b257cfb317ae54fbeb160a40a51df0e84e558ab98ed41040319c0c98e683392"
Sep 19 11:09:37.145544 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.145492    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b257cfb317ae54fbeb160a40a51df0e84e558ab98ed41040319c0c98e683392"} err="failed to get container status \"3b257cfb317ae54fbeb160a40a51df0e84e558ab98ed41040319c0c98e683392\": rpc error: code = NotFound desc = could not find container \"3b257cfb317ae54fbeb160a40a51df0e84e558ab98ed41040319c0c98e683392\": container with ID starting with 3b257cfb317ae54fbeb160a40a51df0e84e558ab98ed41040319c0c98e683392 not found: ID does not exist"
Sep 19 11:09:37.145544 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.145510    2580 scope.go:117] "RemoveContainer" containerID="698a91bfbb3ae66cb81b5ea941b81dd026ebd06ab0dedd1ba5fb237ab522d5f9"
Sep 19 11:09:37.155598 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.155577    2580 scope.go:117] "RemoveContainer" containerID="698a91bfbb3ae66cb81b5ea941b81dd026ebd06ab0dedd1ba5fb237ab522d5f9"
Sep 19 11:09:37.155944 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:09:37.155920    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698a91bfbb3ae66cb81b5ea941b81dd026ebd06ab0dedd1ba5fb237ab522d5f9\": container with ID starting with 698a91bfbb3ae66cb81b5ea941b81dd026ebd06ab0dedd1ba5fb237ab522d5f9 not found: ID does not exist" containerID="698a91bfbb3ae66cb81b5ea941b81dd026ebd06ab0dedd1ba5fb237ab522d5f9"
Sep 19 11:09:37.156023 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.155956    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698a91bfbb3ae66cb81b5ea941b81dd026ebd06ab0dedd1ba5fb237ab522d5f9"} err="failed to get container status \"698a91bfbb3ae66cb81b5ea941b81dd026ebd06ab0dedd1ba5fb237ab522d5f9\": rpc error: code = NotFound desc = could not find container \"698a91bfbb3ae66cb81b5ea941b81dd026ebd06ab0dedd1ba5fb237ab522d5f9\": container with ID starting with 698a91bfbb3ae66cb81b5ea941b81dd026ebd06ab0dedd1ba5fb237ab522d5f9 not found: ID does not exist"
Sep 19 11:09:37.156023 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.155978    2580 scope.go:117] "RemoveContainer" containerID="db342385c3f231b3f67f7cd5f256eb3b46a9076fc38ba36f9583880128a431f3"
Sep 19 11:09:37.159297 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.159273    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f168f4fb-925d-467f-80aa-27c3e74a7093-openshift-service-ca\") pod \"f168f4fb-925d-467f-80aa-27c3e74a7093\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") "
Sep 19 11:09:37.159402 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.159313    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-maas-api-tls\") pod \"f168f4fb-925d-467f-80aa-27c3e74a7093\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") "
Sep 19 11:09:37.159402 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.159362    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-metrics-certs\") pod \"f168f4fb-925d-467f-80aa-27c3e74a7093\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") "
Sep 19 11:09:37.159529 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.159399    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d1f3b1-5768-49e2-abae-5231a55fcf85-plugins-config-volume\") pod \"25d1f3b1-5768-49e2-abae-5231a55fcf85\" (UID: \"25d1f3b1-5768-49e2-abae-5231a55fcf85\") "
Sep 19 11:09:37.159529 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.159425    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8pzx\" (UniqueName: \"kubernetes.io/projected/25d1f3b1-5768-49e2-abae-5231a55fcf85-kube-api-access-w8pzx\") pod \"25d1f3b1-5768-49e2-abae-5231a55fcf85\" (UID: \"25d1f3b1-5768-49e2-abae-5231a55fcf85\") "
Sep 19 11:09:37.159529 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.159493    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pggrm\" (UniqueName: \"kubernetes.io/projected/f168f4fb-925d-467f-80aa-27c3e74a7093-kube-api-access-pggrm\") pod \"f168f4fb-925d-467f-80aa-27c3e74a7093\" (UID: \"f168f4fb-925d-467f-80aa-27c3e74a7093\") "
Sep 19 11:09:37.160878 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.159899    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f168f4fb-925d-467f-80aa-27c3e74a7093-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f168f4fb-925d-467f-80aa-27c3e74a7093" (UID: "f168f4fb-925d-467f-80aa-27c3e74a7093"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:09:37.160878 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.160317    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d1f3b1-5768-49e2-abae-5231a55fcf85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25d1f3b1-5768-49e2-abae-5231a55fcf85" (UID: "25d1f3b1-5768-49e2-abae-5231a55fcf85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:09:37.163131 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.163095    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f168f4fb-925d-467f-80aa-27c3e74a7093-kube-api-access-pggrm" (OuterVolumeSpecName: "kube-api-access-pggrm") pod "f168f4fb-925d-467f-80aa-27c3e74a7093" (UID: "f168f4fb-925d-467f-80aa-27c3e74a7093"). InnerVolumeSpecName "kube-api-access-pggrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:09:37.164079 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.164031    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f168f4fb-925d-467f-80aa-27c3e74a7093" (UID: "f168f4fb-925d-467f-80aa-27c3e74a7093"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:09:37.164896 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.164872    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d1f3b1-5768-49e2-abae-5231a55fcf85-kube-api-access-w8pzx" (OuterVolumeSpecName: "kube-api-access-w8pzx") pod "25d1f3b1-5768-49e2-abae-5231a55fcf85" (UID: "25d1f3b1-5768-49e2-abae-5231a55fcf85"). InnerVolumeSpecName "kube-api-access-w8pzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:09:37.165170 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.165147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f168f4fb-925d-467f-80aa-27c3e74a7093" (UID: "f168f4fb-925d-467f-80aa-27c3e74a7093"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:09:37.166298 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.166279    2580 scope.go:117] "RemoveContainer" containerID="db342385c3f231b3f67f7cd5f256eb3b46a9076fc38ba36f9583880128a431f3"
Sep 19 11:09:37.166614 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:09:37.166593    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db342385c3f231b3f67f7cd5f256eb3b46a9076fc38ba36f9583880128a431f3\": container with ID starting with db342385c3f231b3f67f7cd5f256eb3b46a9076fc38ba36f9583880128a431f3 not found: ID does not exist" containerID="db342385c3f231b3f67f7cd5f256eb3b46a9076fc38ba36f9583880128a431f3"
Sep 19 11:09:37.166708 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.166621    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db342385c3f231b3f67f7cd5f256eb3b46a9076fc38ba36f9583880128a431f3"} err="failed to get container status \"db342385c3f231b3f67f7cd5f256eb3b46a9076fc38ba36f9583880128a431f3\": rpc error: code = NotFound desc = could not find container \"db342385c3f231b3f67f7cd5f256eb3b46a9076fc38ba36f9583880128a431f3\": container with ID starting with db342385c3f231b3f67f7cd5f256eb3b46a9076fc38ba36f9583880128a431f3 not found: ID does not exist"
Sep 19 11:09:37.260446 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.260347    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fchmh\" (UniqueName: \"kubernetes.io/projected/440f5528-193d-4066-b3d4-a57e0464ce83-kube-api-access-fchmh\") pod \"440f5528-193d-4066-b3d4-a57e0464ce83\" (UID: \"440f5528-193d-4066-b3d4-a57e0464ce83\") "
Sep 19 11:09:37.260446 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.260400    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440f5528-193d-4066-b3d4-a57e0464ce83-plugins-config-volume\") pod \"440f5528-193d-4066-b3d4-a57e0464ce83\" (UID: \"440f5528-193d-4066-b3d4-a57e0464ce83\") "
Sep 19 11:09:37.260694 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.260631    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pggrm\" (UniqueName: \"kubernetes.io/projected/f168f4fb-925d-467f-80aa-27c3e74a7093-kube-api-access-pggrm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:37.260694 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.260644    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f168f4fb-925d-467f-80aa-27c3e74a7093-openshift-service-ca\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:37.260694 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.260653    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-maas-api-tls\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:37.260694 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.260666    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f168f4fb-925d-467f-80aa-27c3e74a7093-metrics-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:37.260694 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.260680    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d1f3b1-5768-49e2-abae-5231a55fcf85-plugins-config-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:37.260694 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.260688    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8pzx\" (UniqueName: \"kubernetes.io/projected/25d1f3b1-5768-49e2-abae-5231a55fcf85-kube-api-access-w8pzx\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:37.261040 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.260867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440f5528-193d-4066-b3d4-a57e0464ce83-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "440f5528-193d-4066-b3d4-a57e0464ce83" (UID: "440f5528-193d-4066-b3d4-a57e0464ce83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:09:37.262691 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.262666    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440f5528-193d-4066-b3d4-a57e0464ce83-kube-api-access-fchmh" (OuterVolumeSpecName: "kube-api-access-fchmh") pod "440f5528-193d-4066-b3d4-a57e0464ce83" (UID: "440f5528-193d-4066-b3d4-a57e0464ce83"). InnerVolumeSpecName "kube-api-access-fchmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:09:37.322882 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.322844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:37.362001 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.361967    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fchmh\" (UniqueName: \"kubernetes.io/projected/440f5528-193d-4066-b3d4-a57e0464ce83-kube-api-access-fchmh\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:37.362001 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.362001    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440f5528-193d-4066-b3d4-a57e0464ce83-plugins-config-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:37.420620 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.420582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:37.458127 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.458100    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"]
Sep 19 11:09:37.464165 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.464138    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-063e1d-86b65fc4d5-ltcjz"]
Sep 19 11:09:37.473431 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.473408    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"]
Sep 19 11:09:37.476889 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.476866    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-063e1d-5646cb68d-nfvq2"]
Sep 19 11:09:37.488024 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.488001    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"]
Sep 19 11:09:37.493820 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.493799    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-063e1d-67578845c8-km6xl"]
Sep 19 11:09:37.856710 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.856677    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25d1f3b1-5768-49e2-abae-5231a55fcf85" path="/var/lib/kubelet/pods/25d1f3b1-5768-49e2-abae-5231a55fcf85/volumes"
Sep 19 11:09:37.857115 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.857079    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="440f5528-193d-4066-b3d4-a57e0464ce83" path="/var/lib/kubelet/pods/440f5528-193d-4066-b3d4-a57e0464ce83/volumes"
Sep 19 11:09:37.857415 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:37.857401    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f168f4fb-925d-467f-80aa-27c3e74a7093" path="/var/lib/kubelet/pods/f168f4fb-925d-467f-80aa-27c3e74a7093/volumes"
Sep 19 11:09:38.093597 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:38.093499    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:38.727519 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:38.727480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:40.311896 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:40.311860    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:09:43.226364 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:43.226304    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:43.326203 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:43.326155    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:43.456805 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:43.456769    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"]
Sep 19 11:09:43.457028 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:43.457002    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy" containerID="cri-o://c15c72d0062371eaba326b35f8a8c34ea01892c5140ef1cd0ecc08f296969310" gracePeriod=30
Sep 19 11:09:43.483698 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:43.483603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:43.998179 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:43.998144    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 19 11:09:43.998362 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:43.998204    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 11:09:44.478524 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:44.478481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:45.224589 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:45.224557    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:45.825187 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:45.825156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:48.710085 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.710055    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:09:48.864380 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.864293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-data\") pod \"41fe6072-044c-490d-ad9c-30ef7d235b55\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") "
Sep 19 11:09:48.864380 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.864326    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-credential-socket\") pod \"41fe6072-044c-490d-ad9c-30ef7d235b55\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") "
Sep 19 11:09:48.864605 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.864381    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fe6072-044c-490d-ad9c-30ef7d235b55-istiod-ca-cert\") pod \"41fe6072-044c-490d-ad9c-30ef7d235b55\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") "
Sep 19 11:09:48.864605 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.864427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-envoy\") pod \"41fe6072-044c-490d-ad9c-30ef7d235b55\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") "
Sep 19 11:09:48.864605 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.864456    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpcgz\" (UniqueName: \"kubernetes.io/projected/41fe6072-044c-490d-ad9c-30ef7d235b55-kube-api-access-lpcgz\") pod \"41fe6072-044c-490d-ad9c-30ef7d235b55\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") "
Sep 19 11:09:48.864605 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.864490    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-workload-certs\") pod \"41fe6072-044c-490d-ad9c-30ef7d235b55\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") "
Sep 19 11:09:48.864605 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.864514    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-token\") pod \"41fe6072-044c-490d-ad9c-30ef7d235b55\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") "
Sep 19 11:09:48.864605 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.864571    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-data" (OuterVolumeSpecName: "istio-data") pod "41fe6072-044c-490d-ad9c-30ef7d235b55" (UID: "41fe6072-044c-490d-ad9c-30ef7d235b55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:09:48.864940 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.864621    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-workload-socket\") pod \"41fe6072-044c-490d-ad9c-30ef7d235b55\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") "
Sep 19 11:09:48.864940 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.864666    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "41fe6072-044c-490d-ad9c-30ef7d235b55" (UID: "41fe6072-044c-490d-ad9c-30ef7d235b55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:09:48.864940 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.864679    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-podinfo\") pod \"41fe6072-044c-490d-ad9c-30ef7d235b55\" (UID: \"41fe6072-044c-490d-ad9c-30ef7d235b55\") "
Sep 19 11:09:48.865101 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.864976    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-data\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:48.865101 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.864996    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-credential-socket\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:48.865101 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.865030    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41fe6072-044c-490d-ad9c-30ef7d235b55-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "41fe6072-044c-490d-ad9c-30ef7d235b55" (UID: "41fe6072-044c-490d-ad9c-30ef7d235b55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:09:48.865101 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.865037    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "41fe6072-044c-490d-ad9c-30ef7d235b55" (UID: "41fe6072-044c-490d-ad9c-30ef7d235b55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:09:48.865101 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.865053    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "41fe6072-044c-490d-ad9c-30ef7d235b55" (UID: "41fe6072-044c-490d-ad9c-30ef7d235b55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:09:48.867045 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.867022    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "41fe6072-044c-490d-ad9c-30ef7d235b55" (UID: "41fe6072-044c-490d-ad9c-30ef7d235b55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:09:48.867197 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.867176    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "41fe6072-044c-490d-ad9c-30ef7d235b55" (UID: "41fe6072-044c-490d-ad9c-30ef7d235b55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 11:09:48.867197 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.867189    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-token" (OuterVolumeSpecName: "istio-token") pod "41fe6072-044c-490d-ad9c-30ef7d235b55" (UID: "41fe6072-044c-490d-ad9c-30ef7d235b55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:09:48.867283 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.867193    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fe6072-044c-490d-ad9c-30ef7d235b55-kube-api-access-lpcgz" (OuterVolumeSpecName: "kube-api-access-lpcgz") pod "41fe6072-044c-490d-ad9c-30ef7d235b55" (UID: "41fe6072-044c-490d-ad9c-30ef7d235b55"). InnerVolumeSpecName "kube-api-access-lpcgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:09:48.965687 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.965644    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fe6072-044c-490d-ad9c-30ef7d235b55-istiod-ca-cert\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:48.965687 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.965679    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-envoy\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:48.965687 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.965688    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpcgz\" (UniqueName: \"kubernetes.io/projected/41fe6072-044c-490d-ad9c-30ef7d235b55-kube-api-access-lpcgz\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:48.965687 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.965698    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-workload-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:48.965978 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.965707    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-token\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:48.965978 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.965717    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fe6072-044c-490d-ad9c-30ef7d235b55-workload-socket\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:48.965978 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:48.965725    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fe6072-044c-490d-ad9c-30ef7d235b55-istio-podinfo\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:49.201284 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:49.201252    2580 generic.go:356] "Generic (PLEG): container finished" podID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerID="c15c72d0062371eaba326b35f8a8c34ea01892c5140ef1cd0ecc08f296969310" exitCode=0
Sep 19 11:09:49.201438 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:49.201297    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"
Sep 19 11:09:49.201438 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:49.201327    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" event={"ID":"41fe6072-044c-490d-ad9c-30ef7d235b55","Type":"ContainerDied","Data":"c15c72d0062371eaba326b35f8a8c34ea01892c5140ef1cd0ecc08f296969310"}
Sep 19 11:09:49.201438 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:49.201377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d" event={"ID":"41fe6072-044c-490d-ad9c-30ef7d235b55","Type":"ContainerDied","Data":"35f5933f93fef0704f547eae0955a3fcf4ff2349fefc6de081cd6da77de12455"}
Sep 19 11:09:49.201438 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:49.201399    2580 scope.go:117] "RemoveContainer" containerID="c15c72d0062371eaba326b35f8a8c34ea01892c5140ef1cd0ecc08f296969310"
Sep 19 11:09:49.212851 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:49.212834    2580 scope.go:117] "RemoveContainer" containerID="faa42f013c3e431ef115c81b2e21d3c697081e146402b558dbd79c0deee2b815"
Sep 19 11:09:49.221150 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:49.221126    2580 scope.go:117] "RemoveContainer" containerID="c15c72d0062371eaba326b35f8a8c34ea01892c5140ef1cd0ecc08f296969310"
Sep 19 11:09:49.221534 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:09:49.221509    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15c72d0062371eaba326b35f8a8c34ea01892c5140ef1cd0ecc08f296969310\": container with ID starting with c15c72d0062371eaba326b35f8a8c34ea01892c5140ef1cd0ecc08f296969310 not found: ID does not exist" containerID="c15c72d0062371eaba326b35f8a8c34ea01892c5140ef1cd0ecc08f296969310"
Sep 19 11:09:49.221631 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:49.221541    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15c72d0062371eaba326b35f8a8c34ea01892c5140ef1cd0ecc08f296969310"} err="failed to get container status \"c15c72d0062371eaba326b35f8a8c34ea01892c5140ef1cd0ecc08f296969310\": rpc error: code = NotFound desc = could not find container \"c15c72d0062371eaba326b35f8a8c34ea01892c5140ef1cd0ecc08f296969310\": container with ID starting with c15c72d0062371eaba326b35f8a8c34ea01892c5140ef1cd0ecc08f296969310 not found: ID does not exist"
Sep 19 11:09:49.221631 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:49.221561    2580 scope.go:117] "RemoveContainer" containerID="faa42f013c3e431ef115c81b2e21d3c697081e146402b558dbd79c0deee2b815"
Sep 19 11:09:49.221913 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:09:49.221892    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa42f013c3e431ef115c81b2e21d3c697081e146402b558dbd79c0deee2b815\": container with ID starting with faa42f013c3e431ef115c81b2e21d3c697081e146402b558dbd79c0deee2b815 not found: ID does not exist" containerID="faa42f013c3e431ef115c81b2e21d3c697081e146402b558dbd79c0deee2b815"
Sep 19 11:09:49.221978 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:49.221944    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa42f013c3e431ef115c81b2e21d3c697081e146402b558dbd79c0deee2b815"} err="failed to get container status \"faa42f013c3e431ef115c81b2e21d3c697081e146402b558dbd79c0deee2b815\": rpc error: code = NotFound desc = could not find container \"faa42f013c3e431ef115c81b2e21d3c697081e146402b558dbd79c0deee2b815\": container with ID starting with faa42f013c3e431ef115c81b2e21d3c697081e146402b558dbd79c0deee2b815 not found: ID does not exist"
Sep 19 11:09:49.223149 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:49.223130    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"]
Sep 19 11:09:49.226071 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:49.226051    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-063e1d-openshift-default-6f7dc9ff94-mcs6d"]
Sep 19 11:09:49.856342 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:49.856304    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" path="/var/lib/kubelet/pods/41fe6072-044c-490d-ad9c-30ef7d235b55/volumes"
Sep 19 11:09:51.642001 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:51.641965    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:51.688557 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:51.688524    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:54.242468 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.242436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:54.263947 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.263914    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"]
Sep 19 11:09:54.264188 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.264162    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk" podUID="4d5432bb-551f-4efa-8234-686269667977" containerName="payload-processing" containerID="cri-o://4cfad6b28c305d28feca8152d2e85b2aa3ef046956d7f41dec45cf676a4e4f4d" gracePeriod=30
Sep 19 11:09:54.340229 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.340195    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:54.419123 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.419074    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:54.474692 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.474652    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:54.517413 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.517388    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"
Sep 19 11:09:54.614764 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.614720    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5432bb-551f-4efa-8234-686269667977-plugins-config-volume\") pod \"4d5432bb-551f-4efa-8234-686269667977\" (UID: \"4d5432bb-551f-4efa-8234-686269667977\") "
Sep 19 11:09:54.614934 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.614804    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf964\" (UniqueName: \"kubernetes.io/projected/4d5432bb-551f-4efa-8234-686269667977-kube-api-access-cf964\") pod \"4d5432bb-551f-4efa-8234-686269667977\" (UID: \"4d5432bb-551f-4efa-8234-686269667977\") "
Sep 19 11:09:54.615141 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.615114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5432bb-551f-4efa-8234-686269667977-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d5432bb-551f-4efa-8234-686269667977" (UID: "4d5432bb-551f-4efa-8234-686269667977"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:09:54.617076 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.617053    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5432bb-551f-4efa-8234-686269667977-kube-api-access-cf964" (OuterVolumeSpecName: "kube-api-access-cf964") pod "4d5432bb-551f-4efa-8234-686269667977" (UID: "4d5432bb-551f-4efa-8234-686269667977"). InnerVolumeSpecName "kube-api-access-cf964". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:09:54.715611 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.715555    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5432bb-551f-4efa-8234-686269667977-plugins-config-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:54.715611 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.715604    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf964\" (UniqueName: \"kubernetes.io/projected/4d5432bb-551f-4efa-8234-686269667977-kube-api-access-cf964\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:09:54.779774 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.779662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:54.880696 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:54.880659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:55.239103 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:55.239073    2580 generic.go:356] "Generic (PLEG): container finished" podID="4d5432bb-551f-4efa-8234-686269667977" containerID="4cfad6b28c305d28feca8152d2e85b2aa3ef046956d7f41dec45cf676a4e4f4d" exitCode=0
Sep 19 11:09:55.239288 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:55.239109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk" event={"ID":"4d5432bb-551f-4efa-8234-686269667977","Type":"ContainerDied","Data":"4cfad6b28c305d28feca8152d2e85b2aa3ef046956d7f41dec45cf676a4e4f4d"}
Sep 19 11:09:55.239288 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:55.239116    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"
Sep 19 11:09:55.239288 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:55.239147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk" event={"ID":"4d5432bb-551f-4efa-8234-686269667977","Type":"ContainerDied","Data":"dcda87da6c44c017736e61c6e3c33cf51dbd638acd01e86e376652da994bfba3"}
Sep 19 11:09:55.239288 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:55.239169    2580 scope.go:117] "RemoveContainer" containerID="4cfad6b28c305d28feca8152d2e85b2aa3ef046956d7f41dec45cf676a4e4f4d"
Sep 19 11:09:55.248200 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:55.248180    2580 scope.go:117] "RemoveContainer" containerID="4cfad6b28c305d28feca8152d2e85b2aa3ef046956d7f41dec45cf676a4e4f4d"
Sep 19 11:09:55.248463 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:09:55.248443    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cfad6b28c305d28feca8152d2e85b2aa3ef046956d7f41dec45cf676a4e4f4d\": container with ID starting with 4cfad6b28c305d28feca8152d2e85b2aa3ef046956d7f41dec45cf676a4e4f4d not found: ID does not exist" containerID="4cfad6b28c305d28feca8152d2e85b2aa3ef046956d7f41dec45cf676a4e4f4d"
Sep 19 11:09:55.248508 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:55.248471    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cfad6b28c305d28feca8152d2e85b2aa3ef046956d7f41dec45cf676a4e4f4d"} err="failed to get container status \"4cfad6b28c305d28feca8152d2e85b2aa3ef046956d7f41dec45cf676a4e4f4d\": rpc error: code = NotFound desc = could not find container \"4cfad6b28c305d28feca8152d2e85b2aa3ef046956d7f41dec45cf676a4e4f4d\": container with ID starting with 4cfad6b28c305d28feca8152d2e85b2aa3ef046956d7f41dec45cf676a4e4f4d not found: ID does not exist"
Sep 19 11:09:55.259687 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:55.259660    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"]
Sep 19 11:09:55.263633 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:55.263607    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-137f88-544dd85b8f-crzsk"]
Sep 19 11:09:55.606484 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:55.606395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:09:55.855519 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:55.855485    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d5432bb-551f-4efa-8234-686269667977" path="/var/lib/kubelet/pods/4d5432bb-551f-4efa-8234-686269667977/volumes"
Sep 19 11:09:57.108476 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:09:57.108435    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:10:00.599617 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:00.599585    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:00.699669 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:00.699638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:00.861064 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:00.860964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:01.204808 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.204768    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9"]
Sep 19 11:10:01.205129 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205116    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f168f4fb-925d-467f-80aa-27c3e74a7093" containerName="maas-api"
Sep 19 11:10:01.205187 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205132    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f168f4fb-925d-467f-80aa-27c3e74a7093" containerName="maas-api"
Sep 19 11:10:01.205187 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205144    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440f5528-193d-4066-b3d4-a57e0464ce83" containerName="payload-pre-processing"
Sep 19 11:10:01.205187 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205150    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="440f5528-193d-4066-b3d4-a57e0464ce83" containerName="payload-pre-processing"
Sep 19 11:10:01.205187 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205157    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f9153f-a907-4e31-a1f9-77766b5b6ce7" containerName="main"
Sep 19 11:10:01.205187 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205164    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f9153f-a907-4e31-a1f9-77766b5b6ce7" containerName="main"
Sep 19 11:10:01.205187 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205172    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25d1f3b1-5768-49e2-abae-5231a55fcf85" containerName="payload-processing"
Sep 19 11:10:01.205187 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205180    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="25d1f3b1-5768-49e2-abae-5231a55fcf85" containerName="payload-processing"
Sep 19 11:10:01.205423 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205190    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy"
Sep 19 11:10:01.205423 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205196    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy"
Sep 19 11:10:01.205423 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205207    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy"
Sep 19 11:10:01.205423 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205212    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy"
Sep 19 11:10:01.205423 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205218    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5432bb-551f-4efa-8234-686269667977" containerName="payload-processing"
Sep 19 11:10:01.205423 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205223    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5432bb-551f-4efa-8234-686269667977" containerName="payload-processing"
Sep 19 11:10:01.205423 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205279    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy"
Sep 19 11:10:01.205423 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205287    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="41fe6072-044c-490d-ad9c-30ef7d235b55" containerName="istio-proxy"
Sep 19 11:10:01.205423 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205296    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="48f9153f-a907-4e31-a1f9-77766b5b6ce7" containerName="main"
Sep 19 11:10:01.205423 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205302    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d5432bb-551f-4efa-8234-686269667977" containerName="payload-processing"
Sep 19 11:10:01.205423 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205307    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f168f4fb-925d-467f-80aa-27c3e74a7093" containerName="maas-api"
Sep 19 11:10:01.205423 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205312    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="25d1f3b1-5768-49e2-abae-5231a55fcf85" containerName="payload-processing"
Sep 19 11:10:01.205423 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.205318    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="440f5528-193d-4066-b3d4-a57e0464ce83" containerName="payload-pre-processing"
Sep 19 11:10:01.208268 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.208252    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9"
Sep 19 11:10:01.210144 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.210127    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r42s\""
Sep 19 11:10:01.210230 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.210170    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 11:10:01.215191 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.215166    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9"]
Sep 19 11:10:01.329557 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.329517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:01.374016 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.373978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c854f30-f6d0-4d76-9e85-c077c2207d15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9\" (UID: \"3c854f30-f6d0-4d76-9e85-c077c2207d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9"
Sep 19 11:10:01.475219 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.475130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c854f30-f6d0-4d76-9e85-c077c2207d15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9\" (UID: \"3c854f30-f6d0-4d76-9e85-c077c2207d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9"
Sep 19 11:10:01.475736 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.475716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c854f30-f6d0-4d76-9e85-c077c2207d15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9\" (UID: \"3c854f30-f6d0-4d76-9e85-c077c2207d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9"
Sep 19 11:10:01.519230 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.519191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9"
Sep 19 11:10:01.789243 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:01.789202    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9"]
Sep 19 11:10:01.792123 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:10:01.792098    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c854f30_f6d0_4d76_9e85_c077c2207d15.slice/crio-ed06d22f5ade61642a88cead10f4ff2ce4b6003d0adfda1b06f39aef9304fa0d WatchSource:0}: Error finding container ed06d22f5ade61642a88cead10f4ff2ce4b6003d0adfda1b06f39aef9304fa0d: Status 404 returned error can't find the container with id ed06d22f5ade61642a88cead10f4ff2ce4b6003d0adfda1b06f39aef9304fa0d
Sep 19 11:10:02.120418 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:02.120331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:02.274048 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:02.274009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9" event={"ID":"3c854f30-f6d0-4d76-9e85-c077c2207d15","Type":"ContainerStarted","Data":"3d6c63fea616a324a361b2058237bcfa108750b69a696784ea59085139da4e0d"}
Sep 19 11:10:02.274217 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:02.274071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9" event={"ID":"3c854f30-f6d0-4d76-9e85-c077c2207d15","Type":"ContainerStarted","Data":"ed06d22f5ade61642a88cead10f4ff2ce4b6003d0adfda1b06f39aef9304fa0d"}
Sep 19 11:10:02.295836 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:02.295743    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9" podStartSLOduration=1.295723743 podStartE2EDuration="1.295723743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:10:02.295306737 +0000 UTC m=+2046.954051909" watchObservedRunningTime="2026-09-19 11:10:02.295723743 +0000 UTC m=+2046.954468905"
Sep 19 11:10:02.785568 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:02.785479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:03.284195 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:03.284163    2580 generic.go:356] "Generic (PLEG): container finished" podID="3c854f30-f6d0-4d76-9e85-c077c2207d15" containerID="3d6c63fea616a324a361b2058237bcfa108750b69a696784ea59085139da4e0d" exitCode=0
Sep 19 11:10:03.284590 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:03.284209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9" event={"ID":"3c854f30-f6d0-4d76-9e85-c077c2207d15","Type":"ContainerDied","Data":"3d6c63fea616a324a361b2058237bcfa108750b69a696784ea59085139da4e0d"}
Sep 19 11:10:04.421852 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:04.421826    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9"
Sep 19 11:10:04.601919 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:04.601824    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c854f30-f6d0-4d76-9e85-c077c2207d15-maas-api-service-ca\") pod \"3c854f30-f6d0-4d76-9e85-c077c2207d15\" (UID: \"3c854f30-f6d0-4d76-9e85-c077c2207d15\") "
Sep 19 11:10:04.602207 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:04.602178    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c854f30-f6d0-4d76-9e85-c077c2207d15-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c854f30-f6d0-4d76-9e85-c077c2207d15" (UID: "3c854f30-f6d0-4d76-9e85-c077c2207d15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:10:04.703030 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:04.702998    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c854f30-f6d0-4d76-9e85-c077c2207d15-maas-api-service-ca\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:10:05.295117 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:05.295081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9" event={"ID":"3c854f30-f6d0-4d76-9e85-c077c2207d15","Type":"ContainerDied","Data":"ed06d22f5ade61642a88cead10f4ff2ce4b6003d0adfda1b06f39aef9304fa0d"}
Sep 19 11:10:05.295117 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:05.295114    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9"
Sep 19 11:10:05.295328 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:05.295135    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed06d22f5ade61642a88cead10f4ff2ce4b6003d0adfda1b06f39aef9304fa0d"
Sep 19 11:10:11.721427 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:11.721354    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb"]
Sep 19 11:10:11.723396 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:11.723369    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9b3bef-29830260-66trb"]
Sep 19 11:10:11.723709 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:11.723682    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:11.740111 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:11.740078    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"]
Sep 19 11:10:11.740475 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:11.740438    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt" podUID="1c4607f2-ca97-44e1-92e9-d874d59b2e5e" containerName="payload-processing" containerID="cri-o://382737a9128eec382951bbfb0d5a07597d2b30e7202995686e54149049d03ffe" gracePeriod=30
Sep 19 11:10:11.788363 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:11.788327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:11.856698 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:11.856572    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f085bb5-808c-42d6-81da-cfdf6c8aa4b2" path="/var/lib/kubelet/pods/4f085bb5-808c-42d6-81da-cfdf6c8aa4b2/volumes"
Sep 19 11:10:11.885933 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:11.885893    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9"]
Sep 19 11:10:11.887924 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:11.887898    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9b3bef-9bqh9"]
Sep 19 11:10:11.888269 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:11.888242    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:11.976594 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:11.976479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:12.002351 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.002326    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"
Sep 19 11:10:12.056583 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.056550    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4607f2-ca97-44e1-92e9-d874d59b2e5e-plugins-config-volume\") pod \"1c4607f2-ca97-44e1-92e9-d874d59b2e5e\" (UID: \"1c4607f2-ca97-44e1-92e9-d874d59b2e5e\") "
Sep 19 11:10:12.056772 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.056600    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjznw\" (UniqueName: \"kubernetes.io/projected/1c4607f2-ca97-44e1-92e9-d874d59b2e5e-kube-api-access-pjznw\") pod \"1c4607f2-ca97-44e1-92e9-d874d59b2e5e\" (UID: \"1c4607f2-ca97-44e1-92e9-d874d59b2e5e\") "
Sep 19 11:10:12.056991 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.056966    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4607f2-ca97-44e1-92e9-d874d59b2e5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c4607f2-ca97-44e1-92e9-d874d59b2e5e" (UID: "1c4607f2-ca97-44e1-92e9-d874d59b2e5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:10:12.059214 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.059182    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4607f2-ca97-44e1-92e9-d874d59b2e5e-kube-api-access-pjznw" (OuterVolumeSpecName: "kube-api-access-pjznw") pod "1c4607f2-ca97-44e1-92e9-d874d59b2e5e" (UID: "1c4607f2-ca97-44e1-92e9-d874d59b2e5e"). InnerVolumeSpecName "kube-api-access-pjznw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:10:12.157912 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.157867    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjznw\" (UniqueName: \"kubernetes.io/projected/1c4607f2-ca97-44e1-92e9-d874d59b2e5e-kube-api-access-pjznw\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:10:12.157912 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.157897    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4607f2-ca97-44e1-92e9-d874d59b2e5e-plugins-config-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:10:12.260349 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.260261    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:12.322896 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.322862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:12.328467 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.328440    2580 generic.go:356] "Generic (PLEG): container finished" podID="1c4607f2-ca97-44e1-92e9-d874d59b2e5e" containerID="382737a9128eec382951bbfb0d5a07597d2b30e7202995686e54149049d03ffe" exitCode=0
Sep 19 11:10:12.328600 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.328482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt" event={"ID":"1c4607f2-ca97-44e1-92e9-d874d59b2e5e","Type":"ContainerDied","Data":"382737a9128eec382951bbfb0d5a07597d2b30e7202995686e54149049d03ffe"}
Sep 19 11:10:12.328600 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.328486    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"
Sep 19 11:10:12.328600 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.328523    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt" event={"ID":"1c4607f2-ca97-44e1-92e9-d874d59b2e5e","Type":"ContainerDied","Data":"1c21bf434e10076ec5c93ad1456cfd4ab3aeea983d69329994e1cf31bb6758f6"}
Sep 19 11:10:12.328600 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.328547    2580 scope.go:117] "RemoveContainer" containerID="382737a9128eec382951bbfb0d5a07597d2b30e7202995686e54149049d03ffe"
Sep 19 11:10:12.337574 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.337555    2580 scope.go:117] "RemoveContainer" containerID="382737a9128eec382951bbfb0d5a07597d2b30e7202995686e54149049d03ffe"
Sep 19 11:10:12.337860 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:10:12.337844    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382737a9128eec382951bbfb0d5a07597d2b30e7202995686e54149049d03ffe\": container with ID starting with 382737a9128eec382951bbfb0d5a07597d2b30e7202995686e54149049d03ffe not found: ID does not exist" containerID="382737a9128eec382951bbfb0d5a07597d2b30e7202995686e54149049d03ffe"
Sep 19 11:10:12.337928 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.337868    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382737a9128eec382951bbfb0d5a07597d2b30e7202995686e54149049d03ffe"} err="failed to get container status \"382737a9128eec382951bbfb0d5a07597d2b30e7202995686e54149049d03ffe\": rpc error: code = NotFound desc = could not find container \"382737a9128eec382951bbfb0d5a07597d2b30e7202995686e54149049d03ffe\": container with ID starting with 382737a9128eec382951bbfb0d5a07597d2b30e7202995686e54149049d03ffe not found: ID does not exist"
Sep 19 11:10:12.347792 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.347747    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"]
Sep 19 11:10:12.351897 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:12.351873    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9b3bef-74cbf76ff9-pqdqt"]
Sep 19 11:10:13.856224 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:13.856182    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c4607f2-ca97-44e1-92e9-d874d59b2e5e" path="/var/lib/kubelet/pods/1c4607f2-ca97-44e1-92e9-d874d59b2e5e/volumes"
Sep 19 11:10:13.856726 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:13.856635    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c854f30-f6d0-4d76-9e85-c077c2207d15" path="/var/lib/kubelet/pods/3c854f30-f6d0-4d76-9e85-c077c2207d15/volumes"
Sep 19 11:10:14.413213 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:14.413172    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:10:14.624882 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:14.624848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:14.725658 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:14.725626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:18.198114 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:18.198023    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:18.257640 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:18.257597    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:18.451364 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:18.451266    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:10:23.008522 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:23.008485    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:10:55.961119 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:55.961084    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 11:10:55.962850 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:55.962830    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 11:10:56.217463 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:10:56.217367    2580 scope.go:117] "RemoveContainer" containerID="a2158e6760763350a04efc9b5003b32a7dedc35bc5706c4633defef74bca3e8e"
Sep 19 11:11:44.113557 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:11:44.113523    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:11:45.105357 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:11:45.105321    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:11:46.655508 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:11:46.655472    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:12:24.307297 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:12:24.307258    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:12:32.211898 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:12:32.211848    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:12:40.116548 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:12:40.116508    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:12:55.813007 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:12:55.812974    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:13:01.604705 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:01.604670    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:13:03.301382 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:03.301336    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"]
Sep 19 11:13:03.301915 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:03.301694    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z" podUID="727bbfa8-e3c8-4f40-b05f-f31161f3b1ad" containerName="manager" containerID="cri-o://9f9e90a6f41a8d0acc343900e7db93b8a94b182bd4e6144118babc227e870610" gracePeriod=10
Sep 19 11:13:05.347777 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:05.347735    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 11:13:05.439682 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:05.439641    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/727bbfa8-e3c8-4f40-b05f-f31161f3b1ad-extensions-socket-volume\") pod \"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad\" (UID: \"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad\") "
Sep 19 11:13:05.439682 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:05.439687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw4wb\" (UniqueName: \"kubernetes.io/projected/727bbfa8-e3c8-4f40-b05f-f31161f3b1ad-kube-api-access-pw4wb\") pod \"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad\" (UID: \"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad\") "
Sep 19 11:13:05.440104 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:05.440079    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727bbfa8-e3c8-4f40-b05f-f31161f3b1ad-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "727bbfa8-e3c8-4f40-b05f-f31161f3b1ad" (UID: "727bbfa8-e3c8-4f40-b05f-f31161f3b1ad"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:13:05.441882 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:05.441862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727bbfa8-e3c8-4f40-b05f-f31161f3b1ad-kube-api-access-pw4wb" (OuterVolumeSpecName: "kube-api-access-pw4wb") pod "727bbfa8-e3c8-4f40-b05f-f31161f3b1ad" (UID: "727bbfa8-e3c8-4f40-b05f-f31161f3b1ad"). InnerVolumeSpecName "kube-api-access-pw4wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:13:05.540321 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:05.540281    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/727bbfa8-e3c8-4f40-b05f-f31161f3b1ad-extensions-socket-volume\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:13:05.540321 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:05.540314    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw4wb\" (UniqueName: \"kubernetes.io/projected/727bbfa8-e3c8-4f40-b05f-f31161f3b1ad-kube-api-access-pw4wb\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:13:06.142792 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:06.142682    2580 generic.go:356] "Generic (PLEG): container finished" podID="727bbfa8-e3c8-4f40-b05f-f31161f3b1ad" containerID="9f9e90a6f41a8d0acc343900e7db93b8a94b182bd4e6144118babc227e870610" exitCode=0
Sep 19 11:13:06.142792 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:06.142728    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"
Sep 19 11:13:06.142792 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:06.142782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z" event={"ID":"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad","Type":"ContainerDied","Data":"9f9e90a6f41a8d0acc343900e7db93b8a94b182bd4e6144118babc227e870610"}
Sep 19 11:13:06.143016 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:06.142838    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z" event={"ID":"727bbfa8-e3c8-4f40-b05f-f31161f3b1ad","Type":"ContainerDied","Data":"9c23c8c2d038ce27b76c60775f4cf550e961380b3742647e49dac10b46bb31a4"}
Sep 19 11:13:06.143016 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:06.142860    2580 scope.go:117] "RemoveContainer" containerID="9f9e90a6f41a8d0acc343900e7db93b8a94b182bd4e6144118babc227e870610"
Sep 19 11:13:06.151737 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:06.151716    2580 scope.go:117] "RemoveContainer" containerID="3ba3c1413698a10426615ca08fdfd54a10e05d268761f6dc895f30471734780a"
Sep 19 11:13:06.160346 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:06.160328    2580 scope.go:117] "RemoveContainer" containerID="9f9e90a6f41a8d0acc343900e7db93b8a94b182bd4e6144118babc227e870610"
Sep 19 11:13:06.160517 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:06.160495    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"]
Sep 19 11:13:06.160636 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:13:06.160620    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9e90a6f41a8d0acc343900e7db93b8a94b182bd4e6144118babc227e870610\": container with ID starting with 9f9e90a6f41a8d0acc343900e7db93b8a94b182bd4e6144118babc227e870610 not found: ID does not exist" containerID="9f9e90a6f41a8d0acc343900e7db93b8a94b182bd4e6144118babc227e870610"
Sep 19 11:13:06.160678 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:06.160645    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9e90a6f41a8d0acc343900e7db93b8a94b182bd4e6144118babc227e870610"} err="failed to get container status \"9f9e90a6f41a8d0acc343900e7db93b8a94b182bd4e6144118babc227e870610\": rpc error: code = NotFound desc = could not find container \"9f9e90a6f41a8d0acc343900e7db93b8a94b182bd4e6144118babc227e870610\": container with ID starting with 9f9e90a6f41a8d0acc343900e7db93b8a94b182bd4e6144118babc227e870610 not found: ID does not exist"
Sep 19 11:13:06.160678 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:06.160662    2580 scope.go:117] "RemoveContainer" containerID="3ba3c1413698a10426615ca08fdfd54a10e05d268761f6dc895f30471734780a"
Sep 19 11:13:06.160920 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:13:06.160905    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba3c1413698a10426615ca08fdfd54a10e05d268761f6dc895f30471734780a\": container with ID starting with 3ba3c1413698a10426615ca08fdfd54a10e05d268761f6dc895f30471734780a not found: ID does not exist" containerID="3ba3c1413698a10426615ca08fdfd54a10e05d268761f6dc895f30471734780a"
Sep 19 11:13:06.160981 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:06.160922    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba3c1413698a10426615ca08fdfd54a10e05d268761f6dc895f30471734780a"} err="failed to get container status \"3ba3c1413698a10426615ca08fdfd54a10e05d268761f6dc895f30471734780a\": rpc error: code = NotFound desc = could not find container \"3ba3c1413698a10426615ca08fdfd54a10e05d268761f6dc895f30471734780a\": container with ID starting with 3ba3c1413698a10426615ca08fdfd54a10e05d268761f6dc895f30471734780a not found: ID does not exist"
Sep 19 11:13:06.164944 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:06.164919    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c9z6z"]
Sep 19 11:13:07.855325 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:13:07.855291    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="727bbfa8-e3c8-4f40-b05f-f31161f3b1ad" path="/var/lib/kubelet/pods/727bbfa8-e3c8-4f40-b05f-f31161f3b1ad/volumes"
Sep 19 11:14:03.967211 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.967173    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27"]
Sep 19 11:14:03.967657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.967486    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c854f30-f6d0-4d76-9e85-c077c2207d15" containerName="revoke-keys"
Sep 19 11:14:03.967657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.967497    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c854f30-f6d0-4d76-9e85-c077c2207d15" containerName="revoke-keys"
Sep 19 11:14:03.967657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.967510    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c4607f2-ca97-44e1-92e9-d874d59b2e5e" containerName="payload-processing"
Sep 19 11:14:03.967657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.967516    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c4607f2-ca97-44e1-92e9-d874d59b2e5e" containerName="payload-processing"
Sep 19 11:14:03.967657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.967528    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727bbfa8-e3c8-4f40-b05f-f31161f3b1ad" containerName="manager"
Sep 19 11:14:03.967657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.967536    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="727bbfa8-e3c8-4f40-b05f-f31161f3b1ad" containerName="manager"
Sep 19 11:14:03.967657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.967546    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727bbfa8-e3c8-4f40-b05f-f31161f3b1ad" containerName="manager"
Sep 19 11:14:03.967657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.967552    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="727bbfa8-e3c8-4f40-b05f-f31161f3b1ad" containerName="manager"
Sep 19 11:14:03.967657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.967599    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c854f30-f6d0-4d76-9e85-c077c2207d15" containerName="revoke-keys"
Sep 19 11:14:03.967657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.967608    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="727bbfa8-e3c8-4f40-b05f-f31161f3b1ad" containerName="manager"
Sep 19 11:14:03.967657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.967614    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c4607f2-ca97-44e1-92e9-d874d59b2e5e" containerName="payload-processing"
Sep 19 11:14:03.967657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.967619    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="727bbfa8-e3c8-4f40-b05f-f31161f3b1ad" containerName="manager"
Sep 19 11:14:03.970421 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.970406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27"
Sep 19 11:14:03.972587 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.972565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cxgff\""
Sep 19 11:14:03.982286 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:03.982258    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27"]
Sep 19 11:14:04.029659 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:04.029610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f5422cd-c142-4ef7-8881-a10913370d30-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6xs27\" (UID: \"3f5422cd-c142-4ef7-8881-a10913370d30\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27"
Sep 19 11:14:04.029892 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:04.029686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcfq8\" (UniqueName: \"kubernetes.io/projected/3f5422cd-c142-4ef7-8881-a10913370d30-kube-api-access-kcfq8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6xs27\" (UID: \"3f5422cd-c142-4ef7-8881-a10913370d30\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27"
Sep 19 11:14:04.130884 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:04.130840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcfq8\" (UniqueName: \"kubernetes.io/projected/3f5422cd-c142-4ef7-8881-a10913370d30-kube-api-access-kcfq8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6xs27\" (UID: \"3f5422cd-c142-4ef7-8881-a10913370d30\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27"
Sep 19 11:14:04.131143 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:04.130930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f5422cd-c142-4ef7-8881-a10913370d30-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6xs27\" (UID: \"3f5422cd-c142-4ef7-8881-a10913370d30\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27"
Sep 19 11:14:04.131276 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:04.131261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f5422cd-c142-4ef7-8881-a10913370d30-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6xs27\" (UID: \"3f5422cd-c142-4ef7-8881-a10913370d30\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27"
Sep 19 11:14:04.139387 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:04.139360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcfq8\" (UniqueName: \"kubernetes.io/projected/3f5422cd-c142-4ef7-8881-a10913370d30-kube-api-access-kcfq8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6xs27\" (UID: \"3f5422cd-c142-4ef7-8881-a10913370d30\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27"
Sep 19 11:14:04.280352 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:04.280254    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27"
Sep 19 11:14:04.560553 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:04.560524    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27"]
Sep 19 11:14:04.562353 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:14:04.562325    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5422cd_c142_4ef7_8881_a10913370d30.slice/crio-57953fc03f9343b349c66e16dd724aa2a25175a293000d86bd663862b1f6b39c WatchSource:0}: Error finding container 57953fc03f9343b349c66e16dd724aa2a25175a293000d86bd663862b1f6b39c: Status 404 returned error can't find the container with id 57953fc03f9343b349c66e16dd724aa2a25175a293000d86bd663862b1f6b39c
Sep 19 11:14:05.412556 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:05.412514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27" event={"ID":"3f5422cd-c142-4ef7-8881-a10913370d30","Type":"ContainerStarted","Data":"1822a8ee0361921479fa17a40c6eb3af5b0d34ff57401983656233468c3b4653"}
Sep 19 11:14:05.413021 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:05.412581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27" event={"ID":"3f5422cd-c142-4ef7-8881-a10913370d30","Type":"ContainerStarted","Data":"57953fc03f9343b349c66e16dd724aa2a25175a293000d86bd663862b1f6b39c"}
Sep 19 11:14:05.413021 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:05.412619    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27"
Sep 19 11:14:05.435367 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:05.435304    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27" podStartSLOduration=2.435287662 podStartE2EDuration="2.435287662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:14:05.434006344 +0000 UTC m=+2290.092751503" watchObservedRunningTime="2026-09-19 11:14:05.435287662 +0000 UTC m=+2290.094032820"
Sep 19 11:14:16.419744 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:16.419712    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6xs27"
Sep 19 11:14:26.612172 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:26.612133    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:14:34.005740 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:34.005700    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:14:47.817041 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:47.816953    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"]
Sep 19 11:14:47.820573 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:47.820548    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:47.823248 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:47.823229    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-42b08515-kserve-self-signed-certs\""
Sep 19 11:14:47.828721 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:47.828701    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"]
Sep 19 11:14:47.872770 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:47.872728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:14:48.009818 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.009744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-model-cache\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.009998 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.009851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31614c9c-5710-453a-a52f-01de78ee3c29-tls-certs\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.009998 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.009879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-home\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.009998 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.009914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-dshm\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.009998 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.009956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-tmp-dir\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.009998 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.009984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85q6p\" (UniqueName: \"kubernetes.io/projected/31614c9c-5710-453a-a52f-01de78ee3c29-kube-api-access-85q6p\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.110707 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.110619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-model-cache\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.110707 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.110665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31614c9c-5710-453a-a52f-01de78ee3c29-tls-certs\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.110707 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.110692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-home\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.110965 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.110718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-dshm\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.110965 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.110742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-tmp-dir\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.110965 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.110780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85q6p\" (UniqueName: \"kubernetes.io/projected/31614c9c-5710-453a-a52f-01de78ee3c29-kube-api-access-85q6p\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.111114 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.111091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-model-cache\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.111166 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.111111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-home\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.111211 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.111161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-tmp-dir\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.113309 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.113279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-dshm\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.113556 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.113540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31614c9c-5710-453a-a52f-01de78ee3c29-tls-certs\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.117734 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.117713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85q6p\" (UniqueName: \"kubernetes.io/projected/31614c9c-5710-453a-a52f-01de78ee3c29-kube-api-access-85q6p\") pod \"e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") " pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.132770 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.132734    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:48.403590 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.403560    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"]
Sep 19 11:14:48.406062 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:14:48.406034    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31614c9c_5710_453a_a52f_01de78ee3c29.slice/crio-e66b4be25b4145279600a44ab1c445b276a029167dfbc1dadf3ea47a6e2b3f3b WatchSource:0}: Error finding container e66b4be25b4145279600a44ab1c445b276a029167dfbc1dadf3ea47a6e2b3f3b: Status 404 returned error can't find the container with id e66b4be25b4145279600a44ab1c445b276a029167dfbc1dadf3ea47a6e2b3f3b
Sep 19 11:14:48.617512 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.617472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj" event={"ID":"31614c9c-5710-453a-a52f-01de78ee3c29","Type":"ContainerStarted","Data":"d7c1ad640134314c07f837020f51b0437803d351b7f07f131e257c71b08f189a"}
Sep 19 11:14:48.617512 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.617516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj" event={"ID":"31614c9c-5710-453a-a52f-01de78ee3c29","Type":"ContainerStarted","Data":"e66b4be25b4145279600a44ab1c445b276a029167dfbc1dadf3ea47a6e2b3f3b"}
Sep 19 11:14:48.633314 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:48.633237    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj" podStartSLOduration=1.633221571 podStartE2EDuration="1.633221571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:14:48.631695982 +0000 UTC m=+2333.290441142" watchObservedRunningTime="2026-09-19 11:14:48.633221571 +0000 UTC m=+2333.291966734"
Sep 19 11:14:50.216866 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:50.216832    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:14:54.810321 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.810288    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"]
Sep 19 11:14:54.813364 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.813343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.815627 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.815608    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-42b08515-kserve-self-signed-certs\""
Sep 19 11:14:54.822730 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.822701    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"]
Sep 19 11:14:54.838689 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.838656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:14:54.867964 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.867925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-home\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.867964 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.867960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-dshm\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.868171 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.867988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6kkn\" (UniqueName: \"kubernetes.io/projected/e1018c68-0366-4078-84e7-ec6edb50017d-kube-api-access-r6kkn\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.868171 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.868052    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-tmp-dir\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.868171 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.868106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1018c68-0366-4078-84e7-ec6edb50017d-tls-certs\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.868171 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.868136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-model-cache\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.968653 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.968612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-home\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.968653 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.968650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-dshm\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.968947 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.968806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6kkn\" (UniqueName: \"kubernetes.io/projected/e1018c68-0366-4078-84e7-ec6edb50017d-kube-api-access-r6kkn\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.968947 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.968867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-tmp-dir\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.968947 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.968914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1018c68-0366-4078-84e7-ec6edb50017d-tls-certs\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.969119 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.968960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-model-cache\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.969119 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.969056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-home\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.969225 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.969182    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-tmp-dir\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.969283 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.969273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-model-cache\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.971038 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.971017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-dshm\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.971707 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.971684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1018c68-0366-4078-84e7-ec6edb50017d-tls-certs\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:54.975417 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:54.975399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6kkn\" (UniqueName: \"kubernetes.io/projected/e1018c68-0366-4078-84e7-ec6edb50017d-kube-api-access-r6kkn\") pod \"e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") " pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:55.123948 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:55.123856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:14:55.396577 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:55.396493    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"]
Sep 19 11:14:55.399545 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:14:55.399508    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1018c68_0366_4078_84e7_ec6edb50017d.slice/crio-4be6a61a4de7970576d4c7831724d71578d3062f25ba03b8a9835de7a1747b6c WatchSource:0}: Error finding container 4be6a61a4de7970576d4c7831724d71578d3062f25ba03b8a9835de7a1747b6c: Status 404 returned error can't find the container with id 4be6a61a4de7970576d4c7831724d71578d3062f25ba03b8a9835de7a1747b6c
Sep 19 11:14:55.652011 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:55.651908    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd" event={"ID":"e1018c68-0366-4078-84e7-ec6edb50017d","Type":"ContainerStarted","Data":"882c0919034e002b94ece32c89ace83e3ae85c9c00df68d928599bb7cc470c67"}
Sep 19 11:14:55.652011 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:55.651960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd" event={"ID":"e1018c68-0366-4078-84e7-ec6edb50017d","Type":"ContainerStarted","Data":"4be6a61a4de7970576d4c7831724d71578d3062f25ba03b8a9835de7a1747b6c"}
Sep 19 11:14:55.668919 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:55.668866    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd" podStartSLOduration=1.668849883 podStartE2EDuration="1.668849883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:14:55.667189395 +0000 UTC m=+2340.325934556" watchObservedRunningTime="2026-09-19 11:14:55.668849883 +0000 UTC m=+2340.327595042"
Sep 19 11:14:58.133167 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:58.133130    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:58.133636 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:58.133188    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:58.145985 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:58.145960    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:14:58.419800 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:58.419706    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:14:58.681118 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:14:58.681096    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:15:05.124024 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:05.123975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:15:05.124522 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:05.124046    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:15:05.136500 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:05.136474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:15:05.713322 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:05.713289    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:15:20.844217 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:20.844182    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"]
Sep 19 11:15:20.862141 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:20.862107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:15:21.569009 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:21.568978    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"]
Sep 19 11:15:21.648657 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:21.648616    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:15:24.110573 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:24.110540    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:15:33.611264 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:33.611169    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:15:35.865843 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:35.865802    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj" podUID="31614c9c-5710-453a-a52f-01de78ee3c29" containerName="main" containerID="cri-o://d7c1ad640134314c07f837020f51b0437803d351b7f07f131e257c71b08f189a" gracePeriod=45
Sep 19 11:15:36.117260 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.117201    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:15:36.242679 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.242631    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-tmp-dir\") pod \"31614c9c-5710-453a-a52f-01de78ee3c29\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") "
Sep 19 11:15:36.242917 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.242713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-model-cache\") pod \"31614c9c-5710-453a-a52f-01de78ee3c29\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") "
Sep 19 11:15:36.242917 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.242785    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-home\") pod \"31614c9c-5710-453a-a52f-01de78ee3c29\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") "
Sep 19 11:15:36.242917 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.242820    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-dshm\") pod \"31614c9c-5710-453a-a52f-01de78ee3c29\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") "
Sep 19 11:15:36.242917 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.242845    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31614c9c-5710-453a-a52f-01de78ee3c29-tls-certs\") pod \"31614c9c-5710-453a-a52f-01de78ee3c29\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") "
Sep 19 11:15:36.242917 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.242899    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85q6p\" (UniqueName: \"kubernetes.io/projected/31614c9c-5710-453a-a52f-01de78ee3c29-kube-api-access-85q6p\") pod \"31614c9c-5710-453a-a52f-01de78ee3c29\" (UID: \"31614c9c-5710-453a-a52f-01de78ee3c29\") "
Sep 19 11:15:36.243173 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.242990    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31614c9c-5710-453a-a52f-01de78ee3c29" (UID: "31614c9c-5710-453a-a52f-01de78ee3c29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:15:36.243173 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.243042    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-model-cache" (OuterVolumeSpecName: "model-cache") pod "31614c9c-5710-453a-a52f-01de78ee3c29" (UID: "31614c9c-5710-453a-a52f-01de78ee3c29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:15:36.243173 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.243073    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-home" (OuterVolumeSpecName: "home") pod "31614c9c-5710-453a-a52f-01de78ee3c29" (UID: "31614c9c-5710-453a-a52f-01de78ee3c29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:15:36.243173 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.243157    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:36.243173 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.243168    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:36.243335 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.243177    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:36.245267 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.245233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-dshm" (OuterVolumeSpecName: "dshm") pod "31614c9c-5710-453a-a52f-01de78ee3c29" (UID: "31614c9c-5710-453a-a52f-01de78ee3c29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:15:36.245384 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.245272    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31614c9c-5710-453a-a52f-01de78ee3c29-kube-api-access-85q6p" (OuterVolumeSpecName: "kube-api-access-85q6p") pod "31614c9c-5710-453a-a52f-01de78ee3c29" (UID: "31614c9c-5710-453a-a52f-01de78ee3c29"). InnerVolumeSpecName "kube-api-access-85q6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:15:36.245384 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.245289    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31614c9c-5710-453a-a52f-01de78ee3c29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31614c9c-5710-453a-a52f-01de78ee3c29" (UID: "31614c9c-5710-453a-a52f-01de78ee3c29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:15:36.344081 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.344027    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31614c9c-5710-453a-a52f-01de78ee3c29-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:36.344081 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.344073    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31614c9c-5710-453a-a52f-01de78ee3c29-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:36.344081 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.344090    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85q6p\" (UniqueName: \"kubernetes.io/projected/31614c9c-5710-453a-a52f-01de78ee3c29-kube-api-access-85q6p\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:36.589595 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.589541    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd" podUID="e1018c68-0366-4078-84e7-ec6edb50017d" containerName="main" containerID="cri-o://882c0919034e002b94ece32c89ace83e3ae85c9c00df68d928599bb7cc470c67" gracePeriod=45
Sep 19 11:15:36.822168 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.822143    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:15:36.856643 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.856553    2580 generic.go:356] "Generic (PLEG): container finished" podID="31614c9c-5710-453a-a52f-01de78ee3c29" containerID="d7c1ad640134314c07f837020f51b0437803d351b7f07f131e257c71b08f189a" exitCode=0
Sep 19 11:15:36.856643 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.856594    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"
Sep 19 11:15:36.856886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.856646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj" event={"ID":"31614c9c-5710-453a-a52f-01de78ee3c29","Type":"ContainerDied","Data":"d7c1ad640134314c07f837020f51b0437803d351b7f07f131e257c71b08f189a"}
Sep 19 11:15:36.856886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.856690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj" event={"ID":"31614c9c-5710-453a-a52f-01de78ee3c29","Type":"ContainerDied","Data":"e66b4be25b4145279600a44ab1c445b276a029167dfbc1dadf3ea47a6e2b3f3b"}
Sep 19 11:15:36.856886 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.856712    2580 scope.go:117] "RemoveContainer" containerID="d7c1ad640134314c07f837020f51b0437803d351b7f07f131e257c71b08f189a"
Sep 19 11:15:36.858287 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.858266    2580 generic.go:356] "Generic (PLEG): container finished" podID="e1018c68-0366-4078-84e7-ec6edb50017d" containerID="882c0919034e002b94ece32c89ace83e3ae85c9c00df68d928599bb7cc470c67" exitCode=0
Sep 19 11:15:36.858392 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.858293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd" event={"ID":"e1018c68-0366-4078-84e7-ec6edb50017d","Type":"ContainerDied","Data":"882c0919034e002b94ece32c89ace83e3ae85c9c00df68d928599bb7cc470c67"}
Sep 19 11:15:36.858392 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.858328    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"
Sep 19 11:15:36.858505 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.858328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd" event={"ID":"e1018c68-0366-4078-84e7-ec6edb50017d","Type":"ContainerDied","Data":"4be6a61a4de7970576d4c7831724d71578d3062f25ba03b8a9835de7a1747b6c"}
Sep 19 11:15:36.865736 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.865719    2580 scope.go:117] "RemoveContainer" containerID="d7c1ad640134314c07f837020f51b0437803d351b7f07f131e257c71b08f189a"
Sep 19 11:15:36.866005 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:15:36.865985    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c1ad640134314c07f837020f51b0437803d351b7f07f131e257c71b08f189a\": container with ID starting with d7c1ad640134314c07f837020f51b0437803d351b7f07f131e257c71b08f189a not found: ID does not exist" containerID="d7c1ad640134314c07f837020f51b0437803d351b7f07f131e257c71b08f189a"
Sep 19 11:15:36.866313 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.866012    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c1ad640134314c07f837020f51b0437803d351b7f07f131e257c71b08f189a"} err="failed to get container status \"d7c1ad640134314c07f837020f51b0437803d351b7f07f131e257c71b08f189a\": rpc error: code = NotFound desc = could not find container \"d7c1ad640134314c07f837020f51b0437803d351b7f07f131e257c71b08f189a\": container with ID starting with d7c1ad640134314c07f837020f51b0437803d351b7f07f131e257c71b08f189a not found: ID does not exist"
Sep 19 11:15:36.866313 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.866032    2580 scope.go:117] "RemoveContainer" containerID="882c0919034e002b94ece32c89ace83e3ae85c9c00df68d928599bb7cc470c67"
Sep 19 11:15:36.874271 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.874252    2580 scope.go:117] "RemoveContainer" containerID="882c0919034e002b94ece32c89ace83e3ae85c9c00df68d928599bb7cc470c67"
Sep 19 11:15:36.874540 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:15:36.874523    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882c0919034e002b94ece32c89ace83e3ae85c9c00df68d928599bb7cc470c67\": container with ID starting with 882c0919034e002b94ece32c89ace83e3ae85c9c00df68d928599bb7cc470c67 not found: ID does not exist" containerID="882c0919034e002b94ece32c89ace83e3ae85c9c00df68d928599bb7cc470c67"
Sep 19 11:15:36.874611 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.874546    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882c0919034e002b94ece32c89ace83e3ae85c9c00df68d928599bb7cc470c67"} err="failed to get container status \"882c0919034e002b94ece32c89ace83e3ae85c9c00df68d928599bb7cc470c67\": rpc error: code = NotFound desc = could not find container \"882c0919034e002b94ece32c89ace83e3ae85c9c00df68d928599bb7cc470c67\": container with ID starting with 882c0919034e002b94ece32c89ace83e3ae85c9c00df68d928599bb7cc470c67 not found: ID does not exist"
Sep 19 11:15:36.879476 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.879456    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"]
Sep 19 11:15:36.882742 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.882723    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-42b08515-kserve-6d68549c6c-nx7kj"]
Sep 19 11:15:36.948980 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.948942    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-model-cache\") pod \"e1018c68-0366-4078-84e7-ec6edb50017d\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") "
Sep 19 11:15:36.949138 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.948993    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6kkn\" (UniqueName: \"kubernetes.io/projected/e1018c68-0366-4078-84e7-ec6edb50017d-kube-api-access-r6kkn\") pod \"e1018c68-0366-4078-84e7-ec6edb50017d\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") "
Sep 19 11:15:36.949138 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.949013    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-dshm\") pod \"e1018c68-0366-4078-84e7-ec6edb50017d\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") "
Sep 19 11:15:36.949138 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.949034    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-home\") pod \"e1018c68-0366-4078-84e7-ec6edb50017d\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") "
Sep 19 11:15:36.949138 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.949052    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-tmp-dir\") pod \"e1018c68-0366-4078-84e7-ec6edb50017d\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") "
Sep 19 11:15:36.949138 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.949073    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1018c68-0366-4078-84e7-ec6edb50017d-tls-certs\") pod \"e1018c68-0366-4078-84e7-ec6edb50017d\" (UID: \"e1018c68-0366-4078-84e7-ec6edb50017d\") "
Sep 19 11:15:36.949396 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.949272    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1018c68-0366-4078-84e7-ec6edb50017d" (UID: "e1018c68-0366-4078-84e7-ec6edb50017d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:15:36.949457 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.949394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-home" (OuterVolumeSpecName: "home") pod "e1018c68-0366-4078-84e7-ec6edb50017d" (UID: "e1018c68-0366-4078-84e7-ec6edb50017d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:15:36.949457 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.949408    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1018c68-0366-4078-84e7-ec6edb50017d" (UID: "e1018c68-0366-4078-84e7-ec6edb50017d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:15:36.949457 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.949400    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-model-cache\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:36.951567 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.951540    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-dshm" (OuterVolumeSpecName: "dshm") pod "e1018c68-0366-4078-84e7-ec6edb50017d" (UID: "e1018c68-0366-4078-84e7-ec6edb50017d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:15:36.951890 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.951863    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1018c68-0366-4078-84e7-ec6edb50017d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1018c68-0366-4078-84e7-ec6edb50017d" (UID: "e1018c68-0366-4078-84e7-ec6edb50017d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:15:36.951890 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:36.951875    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1018c68-0366-4078-84e7-ec6edb50017d-kube-api-access-r6kkn" (OuterVolumeSpecName: "kube-api-access-r6kkn") pod "e1018c68-0366-4078-84e7-ec6edb50017d" (UID: "e1018c68-0366-4078-84e7-ec6edb50017d"). InnerVolumeSpecName "kube-api-access-r6kkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:15:37.050025 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:37.049989    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6kkn\" (UniqueName: \"kubernetes.io/projected/e1018c68-0366-4078-84e7-ec6edb50017d-kube-api-access-r6kkn\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:37.050025 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:37.050020    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-dshm\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:37.050025 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:37.050032    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-home\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:37.050253 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:37.050040    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1018c68-0366-4078-84e7-ec6edb50017d-tmp-dir\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:37.050253 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:37.050048    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1018c68-0366-4078-84e7-ec6edb50017d-tls-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:37.181293 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:37.181252    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"]
Sep 19 11:15:37.186620 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:37.186593    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-42b08515-kserve-78bd654b87-4b5zd"]
Sep 19 11:15:37.856588 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:37.856556    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31614c9c-5710-453a-a52f-01de78ee3c29" path="/var/lib/kubelet/pods/31614c9c-5710-453a-a52f-01de78ee3c29/volumes"
Sep 19 11:15:37.856986 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:37.856970    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1018c68-0366-4078-84e7-ec6edb50017d" path="/var/lib/kubelet/pods/e1018c68-0366-4078-84e7-ec6edb50017d/volumes"
Sep 19 11:15:41.312219 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.312182    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:15:41.412291 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.412259    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"]
Sep 19 11:15:41.412587 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.412574    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1018c68-0366-4078-84e7-ec6edb50017d" containerName="main"
Sep 19 11:15:41.412635 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.412588    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1018c68-0366-4078-84e7-ec6edb50017d" containerName="main"
Sep 19 11:15:41.412635 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.412603    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31614c9c-5710-453a-a52f-01de78ee3c29" containerName="main"
Sep 19 11:15:41.412635 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.412610    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31614c9c-5710-453a-a52f-01de78ee3c29" containerName="main"
Sep 19 11:15:41.412735 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.412669    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="31614c9c-5710-453a-a52f-01de78ee3c29" containerName="main"
Sep 19 11:15:41.412735 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.412679    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1018c68-0366-4078-84e7-ec6edb50017d" containerName="main"
Sep 19 11:15:41.417093 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.417066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:41.420257 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.420233    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 19 11:15:41.420457 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.420234    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 19 11:15:41.420557 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.420266    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 19 11:15:41.420557 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.420266    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g2pqv\""
Sep 19 11:15:41.420557 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.420323    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 11:15:41.425741 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.425682    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"]
Sep 19 11:15:41.588376 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.588285    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91c52aed-d344-4920-816f-b85a5a656d70-maas-api-tls\") pod \"maas-api-6599cb9dcd-s6wkz\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") " pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:41.588376 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.588342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c52aed-d344-4920-816f-b85a5a656d70-openshift-service-ca\") pod \"maas-api-6599cb9dcd-s6wkz\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") " pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:41.588376 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.588358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91c52aed-d344-4920-816f-b85a5a656d70-metrics-certs\") pod \"maas-api-6599cb9dcd-s6wkz\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") " pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:41.588604 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.588471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxptq\" (UniqueName: \"kubernetes.io/projected/91c52aed-d344-4920-816f-b85a5a656d70-kube-api-access-bxptq\") pod \"maas-api-6599cb9dcd-s6wkz\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") " pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:41.688914 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.688869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c52aed-d344-4920-816f-b85a5a656d70-openshift-service-ca\") pod \"maas-api-6599cb9dcd-s6wkz\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") " pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:41.688914 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.688913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91c52aed-d344-4920-816f-b85a5a656d70-metrics-certs\") pod \"maas-api-6599cb9dcd-s6wkz\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") " pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:41.689151 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.688953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxptq\" (UniqueName: \"kubernetes.io/projected/91c52aed-d344-4920-816f-b85a5a656d70-kube-api-access-bxptq\") pod \"maas-api-6599cb9dcd-s6wkz\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") " pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:41.689151 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.688984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91c52aed-d344-4920-816f-b85a5a656d70-maas-api-tls\") pod \"maas-api-6599cb9dcd-s6wkz\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") " pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:41.689666 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.689643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c52aed-d344-4920-816f-b85a5a656d70-openshift-service-ca\") pod \"maas-api-6599cb9dcd-s6wkz\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") " pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:41.691710 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.691687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91c52aed-d344-4920-816f-b85a5a656d70-metrics-certs\") pod \"maas-api-6599cb9dcd-s6wkz\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") " pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:41.691834 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.691721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91c52aed-d344-4920-816f-b85a5a656d70-maas-api-tls\") pod \"maas-api-6599cb9dcd-s6wkz\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") " pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:41.696000 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.695979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxptq\" (UniqueName: \"kubernetes.io/projected/91c52aed-d344-4920-816f-b85a5a656d70-kube-api-access-bxptq\") pod \"maas-api-6599cb9dcd-s6wkz\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") " pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:41.730193 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:41.730154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:42.003793 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:42.003716    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"]
Sep 19 11:15:42.006455 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:15:42.006421    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c52aed_d344_4920_816f_b85a5a656d70.slice/crio-9adb58825200ab5c38faed6eee8dac8f30c4fb69d778c12433208e007811ca45 WatchSource:0}: Error finding container 9adb58825200ab5c38faed6eee8dac8f30c4fb69d778c12433208e007811ca45: Status 404 returned error can't find the container with id 9adb58825200ab5c38faed6eee8dac8f30c4fb69d778c12433208e007811ca45
Sep 19 11:15:42.007819 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:42.007799    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 11:15:42.889829 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:42.889783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz" event={"ID":"91c52aed-d344-4920-816f-b85a5a656d70","Type":"ContainerStarted","Data":"33ee1df3adb8f6f33382d9b64766dc0b4df4e94a16e5006c171249935b47a585"}
Sep 19 11:15:42.890234 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:42.889843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz" event={"ID":"91c52aed-d344-4920-816f-b85a5a656d70","Type":"ContainerStarted","Data":"9adb58825200ab5c38faed6eee8dac8f30c4fb69d778c12433208e007811ca45"}
Sep 19 11:15:42.890234 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:42.889895    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:42.905331 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:42.905170    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz" podStartSLOduration=1.496821897 podStartE2EDuration="1.905153439s" totalImagesPullingTime="408.331542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:15:42.9044988 +0000 UTC m=+2387.563243961" watchObservedRunningTime="2026-09-19 11:15:42.905153439 +0000 UTC m=+2387.563898601"
Sep 19 11:15:45.646555 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:45.646511    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:15:48.899648 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:48.899620    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:49.286280 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.286241    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"]
Sep 19 11:15:49.289728 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.289708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.297510 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.297418    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"]
Sep 19 11:15:49.360946 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.360905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nxkd\" (UniqueName: \"kubernetes.io/projected/7a1514ec-d9ee-4573-b257-f3f44ac195ee-kube-api-access-5nxkd\") pod \"maas-api-6d474ff8d6-cg7sf\" (UID: \"7a1514ec-d9ee-4573-b257-f3f44ac195ee\") " pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.361128 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.360953    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1514ec-d9ee-4573-b257-f3f44ac195ee-metrics-certs\") pod \"maas-api-6d474ff8d6-cg7sf\" (UID: \"7a1514ec-d9ee-4573-b257-f3f44ac195ee\") " pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.361128 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.361006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1514ec-d9ee-4573-b257-f3f44ac195ee-openshift-service-ca\") pod \"maas-api-6d474ff8d6-cg7sf\" (UID: \"7a1514ec-d9ee-4573-b257-f3f44ac195ee\") " pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.361128 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.361113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1514ec-d9ee-4573-b257-f3f44ac195ee-maas-api-tls\") pod \"maas-api-6d474ff8d6-cg7sf\" (UID: \"7a1514ec-d9ee-4573-b257-f3f44ac195ee\") " pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.462274 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.462240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1514ec-d9ee-4573-b257-f3f44ac195ee-metrics-certs\") pod \"maas-api-6d474ff8d6-cg7sf\" (UID: \"7a1514ec-d9ee-4573-b257-f3f44ac195ee\") " pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.462274 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.462281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1514ec-d9ee-4573-b257-f3f44ac195ee-openshift-service-ca\") pod \"maas-api-6d474ff8d6-cg7sf\" (UID: \"7a1514ec-d9ee-4573-b257-f3f44ac195ee\") " pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.462533 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.462338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1514ec-d9ee-4573-b257-f3f44ac195ee-maas-api-tls\") pod \"maas-api-6d474ff8d6-cg7sf\" (UID: \"7a1514ec-d9ee-4573-b257-f3f44ac195ee\") " pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.462533 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.462404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nxkd\" (UniqueName: \"kubernetes.io/projected/7a1514ec-d9ee-4573-b257-f3f44ac195ee-kube-api-access-5nxkd\") pod \"maas-api-6d474ff8d6-cg7sf\" (UID: \"7a1514ec-d9ee-4573-b257-f3f44ac195ee\") " pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.463088 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.463058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1514ec-d9ee-4573-b257-f3f44ac195ee-openshift-service-ca\") pod \"maas-api-6d474ff8d6-cg7sf\" (UID: \"7a1514ec-d9ee-4573-b257-f3f44ac195ee\") " pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.465065 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.465037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1514ec-d9ee-4573-b257-f3f44ac195ee-metrics-certs\") pod \"maas-api-6d474ff8d6-cg7sf\" (UID: \"7a1514ec-d9ee-4573-b257-f3f44ac195ee\") " pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.465158 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.465142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1514ec-d9ee-4573-b257-f3f44ac195ee-maas-api-tls\") pod \"maas-api-6d474ff8d6-cg7sf\" (UID: \"7a1514ec-d9ee-4573-b257-f3f44ac195ee\") " pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.469399 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.469370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nxkd\" (UniqueName: \"kubernetes.io/projected/7a1514ec-d9ee-4573-b257-f3f44ac195ee-kube-api-access-5nxkd\") pod \"maas-api-6d474ff8d6-cg7sf\" (UID: \"7a1514ec-d9ee-4573-b257-f3f44ac195ee\") " pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.601446 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.601363    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:49.609494 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.609462    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:15:49.873323 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.873291    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"]
Sep 19 11:15:49.875995 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:15:49.875970    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1514ec_d9ee_4573_b257_f3f44ac195ee.slice/crio-952859a2bd343f319f48386a68ea353ca08833e6d69a6a65dd91d25c7189c2fc WatchSource:0}: Error finding container 952859a2bd343f319f48386a68ea353ca08833e6d69a6a65dd91d25c7189c2fc: Status 404 returned error can't find the container with id 952859a2bd343f319f48386a68ea353ca08833e6d69a6a65dd91d25c7189c2fc
Sep 19 11:15:49.922372 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:49.922332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf" event={"ID":"7a1514ec-d9ee-4573-b257-f3f44ac195ee","Type":"ContainerStarted","Data":"952859a2bd343f319f48386a68ea353ca08833e6d69a6a65dd91d25c7189c2fc"}
Sep 19 11:15:50.929013 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:50.928978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf" event={"ID":"7a1514ec-d9ee-4573-b257-f3f44ac195ee","Type":"ContainerStarted","Data":"b26793932b85be44680fc0a13533afd7b49fa4e231e55655797a71339e8908d5"}
Sep 19 11:15:50.929394 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:50.929053    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:50.945338 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:50.945289    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf" podStartSLOduration=1.618786429 podStartE2EDuration="1.945275265s" totalImagesPullingTime="326.488836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:15:50.943390559 +0000 UTC m=+2395.602135718" watchObservedRunningTime="2026-09-19 11:15:50.945275265 +0000 UTC m=+2395.604020425"
Sep 19 11:15:55.990222 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:55.990194    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 11:15:55.993358 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:55.993338    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kx7wh_cadc00c5-c78d-4548-a031-b59af85e69e7/ovn-acl-logging/0.log"
Sep 19 11:15:56.417021 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:56.416979    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:15:56.938847 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:56.938821    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d474ff8d6-cg7sf"
Sep 19 11:15:56.981500 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:56.981463    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"]
Sep 19 11:15:56.981734 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:56.981701    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz" podUID="91c52aed-d344-4920-816f-b85a5a656d70" containerName="maas-api" containerID="cri-o://33ee1df3adb8f6f33382d9b64766dc0b4df4e94a16e5006c171249935b47a585" gracePeriod=30
Sep 19 11:15:57.229888 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.229865    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:57.327514 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.327460    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c52aed-d344-4920-816f-b85a5a656d70-openshift-service-ca\") pod \"91c52aed-d344-4920-816f-b85a5a656d70\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") "
Sep 19 11:15:57.327514 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.327523    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91c52aed-d344-4920-816f-b85a5a656d70-maas-api-tls\") pod \"91c52aed-d344-4920-816f-b85a5a656d70\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") "
Sep 19 11:15:57.327815 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.327561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91c52aed-d344-4920-816f-b85a5a656d70-metrics-certs\") pod \"91c52aed-d344-4920-816f-b85a5a656d70\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") "
Sep 19 11:15:57.327815 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.327609    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxptq\" (UniqueName: \"kubernetes.io/projected/91c52aed-d344-4920-816f-b85a5a656d70-kube-api-access-bxptq\") pod \"91c52aed-d344-4920-816f-b85a5a656d70\" (UID: \"91c52aed-d344-4920-816f-b85a5a656d70\") "
Sep 19 11:15:57.327984 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.327957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c52aed-d344-4920-816f-b85a5a656d70-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91c52aed-d344-4920-816f-b85a5a656d70" (UID: "91c52aed-d344-4920-816f-b85a5a656d70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:15:57.329981 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.329950    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c52aed-d344-4920-816f-b85a5a656d70-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "91c52aed-d344-4920-816f-b85a5a656d70" (UID: "91c52aed-d344-4920-816f-b85a5a656d70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:15:57.329981 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.329970    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c52aed-d344-4920-816f-b85a5a656d70-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91c52aed-d344-4920-816f-b85a5a656d70" (UID: "91c52aed-d344-4920-816f-b85a5a656d70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:15:57.330157 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.330010    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c52aed-d344-4920-816f-b85a5a656d70-kube-api-access-bxptq" (OuterVolumeSpecName: "kube-api-access-bxptq") pod "91c52aed-d344-4920-816f-b85a5a656d70" (UID: "91c52aed-d344-4920-816f-b85a5a656d70"). InnerVolumeSpecName "kube-api-access-bxptq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:15:57.428435 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.428395    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxptq\" (UniqueName: \"kubernetes.io/projected/91c52aed-d344-4920-816f-b85a5a656d70-kube-api-access-bxptq\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:57.428435 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.428427    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c52aed-d344-4920-816f-b85a5a656d70-openshift-service-ca\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:57.428435 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.428437    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91c52aed-d344-4920-816f-b85a5a656d70-maas-api-tls\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:57.428435 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.428445    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91c52aed-d344-4920-816f-b85a5a656d70-metrics-certs\") on node \"ip-10-0-129-220.ec2.internal\" DevicePath \"\""
Sep 19 11:15:57.963107 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.963070    2580 generic.go:356] "Generic (PLEG): container finished" podID="91c52aed-d344-4920-816f-b85a5a656d70" containerID="33ee1df3adb8f6f33382d9b64766dc0b4df4e94a16e5006c171249935b47a585" exitCode=0
Sep 19 11:15:57.963276 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.963126    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"
Sep 19 11:15:57.963276 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.963134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz" event={"ID":"91c52aed-d344-4920-816f-b85a5a656d70","Type":"ContainerDied","Data":"33ee1df3adb8f6f33382d9b64766dc0b4df4e94a16e5006c171249935b47a585"}
Sep 19 11:15:57.963276 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.963182    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz" event={"ID":"91c52aed-d344-4920-816f-b85a5a656d70","Type":"ContainerDied","Data":"9adb58825200ab5c38faed6eee8dac8f30c4fb69d778c12433208e007811ca45"}
Sep 19 11:15:57.963276 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.963204    2580 scope.go:117] "RemoveContainer" containerID="33ee1df3adb8f6f33382d9b64766dc0b4df4e94a16e5006c171249935b47a585"
Sep 19 11:15:57.972347 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.972319    2580 scope.go:117] "RemoveContainer" containerID="33ee1df3adb8f6f33382d9b64766dc0b4df4e94a16e5006c171249935b47a585"
Sep 19 11:15:57.972625 ip-10-0-129-220 kubenswrapper[2580]: E0919 11:15:57.972603    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ee1df3adb8f6f33382d9b64766dc0b4df4e94a16e5006c171249935b47a585\": container with ID starting with 33ee1df3adb8f6f33382d9b64766dc0b4df4e94a16e5006c171249935b47a585 not found: ID does not exist" containerID="33ee1df3adb8f6f33382d9b64766dc0b4df4e94a16e5006c171249935b47a585"
Sep 19 11:15:57.972693 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.972638    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ee1df3adb8f6f33382d9b64766dc0b4df4e94a16e5006c171249935b47a585"} err="failed to get container status \"33ee1df3adb8f6f33382d9b64766dc0b4df4e94a16e5006c171249935b47a585\": rpc error: code = NotFound desc = could not find container \"33ee1df3adb8f6f33382d9b64766dc0b4df4e94a16e5006c171249935b47a585\": container with ID starting with 33ee1df3adb8f6f33382d9b64766dc0b4df4e94a16e5006c171249935b47a585 not found: ID does not exist"
Sep 19 11:15:57.978717 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.978692    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"]
Sep 19 11:15:57.980870 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:57.980848    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599cb9dcd-s6wkz"]
Sep 19 11:15:59.855357 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:15:59.855325    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91c52aed-d344-4920-816f-b85a5a656d70" path="/var/lib/kubelet/pods/91c52aed-d344-4920-816f-b85a5a656d70/volumes"
Sep 19 11:16:02.312667 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:02.312631    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:16:09.310080 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:09.310031    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4th8x"]
Sep 19 11:16:16.929078 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:16.928995    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wwqqp_798963fa-d2b8-45c0-836e-4a8eb892c62e/manager/0.log"
Sep 19 11:16:17.044344 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:17.044314    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6qkpw_f2ab3b71-ec29-4b92-8165-2fd479aba8ca/manager/0.log"
Sep 19 11:16:17.411217 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:17.411184    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-h9wxr_04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc/manager/0.log"
Sep 19 11:16:17.669803 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:17.669697    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt_43a85296-9b77-4c62-b5a5-ab4edc8e3e67/manager/0.log"
Sep 19 11:16:17.674772 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:17.674730    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt_43a85296-9b77-4c62-b5a5-ab4edc8e3e67/copy-manifests/0.log"
Sep 19 11:16:18.241343 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:18.241305    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d474ff8d6-cg7sf_7a1514ec-d9ee-4573-b257-f3f44ac195ee/maas-api/0.log"
Sep 19 11:16:18.355853 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:18.355829    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29830260-wkdqz_6dff9a57-bbb1-42fb-86fe-fd45d0ba32fb/cleanup/0.log"
Sep 19 11:16:19.426393 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:19.426358    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-jnvrj_c3539e5e-710c-4b48-8205-92dd4b67f8ca/manager/0.log"
Sep 19 11:16:19.527512 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:19.527463    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-6ptv6_338f0495-8618-4e40-a96b-b74298ec67d1/manager/0.log"
Sep 19 11:16:19.637890 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:19.637859    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-wzqdr_4fcae889-b4f9-4446-92a5-cc8715472411/kuadrant-console-plugin/0.log"
Sep 19 11:16:19.771337 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:19.771305    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-6xs27_3f5422cd-c142-4ef7-8881-a10913370d30/manager/0.log"
Sep 19 11:16:19.893129 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:19.893094    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-4th8x_55daafdd-2998-45d4-9323-8b438b474d5b/limitador/0.log"
Sep 19 11:16:20.017491 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:20.017454    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-tm6rs_7ffdf181-a443-40c0-89d2-784c52d59871/manager/0.log"
Sep 19 11:16:20.678520 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:20.678484    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d97bd96f8-tghnl_688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa/kube-auth-proxy/0.log"
Sep 19 11:16:21.450989 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:21.450958    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-lhnnc_d3cd7942-80aa-4ee4-86e9-8a5340c414c4/main/0.log"
Sep 19 11:16:21.892609 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:21.892518    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f65q7t_6d16400b-68ba-47f6-9939-2e798850814e/main/0.log"
Sep 19 11:16:22.096890 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:22.096856    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-kt7fj_37dbaf07-ee66-4cb3-b34c-50d772158ac8/main/0.log"
Sep 19 11:16:27.251233 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:27.251198    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:16:27.311234 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:27.311189    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log" limit=1024
Sep 19 11:16:30.395801 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:30.395735    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wxp6f_f24a9314-df59-4a55-8d61-9b6f278c322d/global-pull-secret-syncer/0.log"
Sep 19 11:16:30.444747 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:30.444717    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2wmgv_9d113728-f674-42ed-b2d3-bb9b96e30704/konnectivity-agent/0.log"
Sep 19 11:16:30.536574 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:30.536544    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-220.ec2.internal_4197a5b43e365e5128cbafdb49a967bc/haproxy/0.log"
Sep 19 11:16:35.344824 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:35.344794    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-jnvrj_c3539e5e-710c-4b48-8205-92dd4b67f8ca/manager/0.log"
Sep 19 11:16:35.390961 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:35.390913    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-6ptv6_338f0495-8618-4e40-a96b-b74298ec67d1/manager/0.log"
Sep 19 11:16:35.419619 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:35.419591    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-wzqdr_4fcae889-b4f9-4446-92a5-cc8715472411/kuadrant-console-plugin/0.log"
Sep 19 11:16:35.482664 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:35.482626    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-6xs27_3f5422cd-c142-4ef7-8881-a10913370d30/manager/0.log"
Sep 19 11:16:35.509732 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:35.509705    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-4th8x_55daafdd-2998-45d4-9323-8b438b474d5b/limitador/0.log"
Sep 19 11:16:35.597064 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:35.596940    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-tm6rs_7ffdf181-a443-40c0-89d2-784c52d59871/manager/0.log"
Sep 19 11:16:37.687697 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:37.687666    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6kpzj_f1f7e08f-bcc2-4995-b381-0efaf8682d67/node-exporter/0.log"
Sep 19 11:16:37.711539 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:37.711511    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6kpzj_f1f7e08f-bcc2-4995-b381-0efaf8682d67/kube-rbac-proxy/0.log"
Sep 19 11:16:37.738747 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:37.738721    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6kpzj_f1f7e08f-bcc2-4995-b381-0efaf8682d67/init-textfile/0.log"
Sep 19 11:16:41.429873 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.429835    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"]
Sep 19 11:16:41.430347 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.430158    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91c52aed-d344-4920-816f-b85a5a656d70" containerName="maas-api"
Sep 19 11:16:41.430347 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.430169    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="91c52aed-d344-4920-816f-b85a5a656d70" containerName="maas-api"
Sep 19 11:16:41.430347 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.430228    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="91c52aed-d344-4920-816f-b85a5a656d70" containerName="maas-api"
Sep 19 11:16:41.433109 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.433092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.435368 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.435334    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4wp77\"/\"default-dockercfg-5dxml\""
Sep 19 11:16:41.435504 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.435392    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4wp77\"/\"kube-root-ca.crt\""
Sep 19 11:16:41.436062 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.436036    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4wp77\"/\"openshift-service-ca.crt\""
Sep 19 11:16:41.439720 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.439696    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"]
Sep 19 11:16:41.502968 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.502932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fab4bf51-c4ac-40f9-b8d6-692e8040334b-lib-modules\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.502968 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.502970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fab4bf51-c4ac-40f9-b8d6-692e8040334b-sys\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.503211 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.503007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fab4bf51-c4ac-40f9-b8d6-692e8040334b-proc\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.503211 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.503031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fab4bf51-c4ac-40f9-b8d6-692e8040334b-podres\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.503211 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.503053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9ffs\" (UniqueName: \"kubernetes.io/projected/fab4bf51-c4ac-40f9-b8d6-692e8040334b-kube-api-access-v9ffs\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.603985 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.603944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fab4bf51-c4ac-40f9-b8d6-692e8040334b-lib-modules\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.603985 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.603985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fab4bf51-c4ac-40f9-b8d6-692e8040334b-sys\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.604239 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.604024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fab4bf51-c4ac-40f9-b8d6-692e8040334b-proc\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.604239 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.604038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fab4bf51-c4ac-40f9-b8d6-692e8040334b-podres\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.604239 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.604053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9ffs\" (UniqueName: \"kubernetes.io/projected/fab4bf51-c4ac-40f9-b8d6-692e8040334b-kube-api-access-v9ffs\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.604239 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.604141    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fab4bf51-c4ac-40f9-b8d6-692e8040334b-proc\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.604239 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.604163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fab4bf51-c4ac-40f9-b8d6-692e8040334b-sys\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.604239 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.604141    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fab4bf51-c4ac-40f9-b8d6-692e8040334b-lib-modules\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.604239 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.604163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fab4bf51-c4ac-40f9-b8d6-692e8040334b-podres\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.611487 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.611457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9ffs\" (UniqueName: \"kubernetes.io/projected/fab4bf51-c4ac-40f9-b8d6-692e8040334b-kube-api-access-v9ffs\") pod \"perf-node-gather-daemonset-969qh\" (UID: \"fab4bf51-c4ac-40f9-b8d6-692e8040334b\") " pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:41.743158 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:41.743131    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:42.010229 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:42.010202    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"]
Sep 19 11:16:42.012712 ip-10-0-129-220 kubenswrapper[2580]: W0919 11:16:42.012686    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfab4bf51_c4ac_40f9_b8d6_692e8040334b.slice/crio-beae8a526ee58f124d70efa8c7a1c0b8164c94f1dc8b906418e828f73df1d363 WatchSource:0}: Error finding container beae8a526ee58f124d70efa8c7a1c0b8164c94f1dc8b906418e828f73df1d363: Status 404 returned error can't find the container with id beae8a526ee58f124d70efa8c7a1c0b8164c94f1dc8b906418e828f73df1d363
Sep 19 11:16:42.170827 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:42.170792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh" event={"ID":"fab4bf51-c4ac-40f9-b8d6-692e8040334b","Type":"ContainerStarted","Data":"0f6992bc0b9f843fe35aaf1ecb8b719b785192e4b5417552dab770e1ba6f780e"}
Sep 19 11:16:42.171021 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:42.170853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh" event={"ID":"fab4bf51-c4ac-40f9-b8d6-692e8040334b","Type":"ContainerStarted","Data":"beae8a526ee58f124d70efa8c7a1c0b8164c94f1dc8b906418e828f73df1d363"}
Sep 19 11:16:42.171021 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:42.170893    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:42.184224 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:42.184178    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh" podStartSLOduration=1.18415564 podStartE2EDuration="1.18415564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:16:42.184024038 +0000 UTC m=+2446.842769197" watchObservedRunningTime="2026-09-19 11:16:42.18415564 +0000 UTC m=+2446.842900800"
Sep 19 11:16:43.377040 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:43.377010    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ncb5v_12123b50-d178-4a45-aab4-1785cba2616f/dns/0.log"
Sep 19 11:16:43.396399 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:43.396370    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ncb5v_12123b50-d178-4a45-aab4-1785cba2616f/kube-rbac-proxy/0.log"
Sep 19 11:16:43.509878 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:43.509849    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dkccw_4c1b1206-2361-42b9-b787-b49f81c893c7/dns-node-resolver/0.log"
Sep 19 11:16:44.098492 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:44.098462    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-49h5q_b5bae54d-ce02-46ec-a38c-c43f61441d7d/node-ca/0.log"
Sep 19 11:16:45.564599 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:45.564572    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d97bd96f8-tghnl_688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa/kube-auth-proxy/0.log"
Sep 19 11:16:46.644308 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:46.644274    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mjqjg_97db0dbe-bb49-4a08-99dc-0766b211bfbe/serve-healthcheck-canary/0.log"
Sep 19 11:16:47.132948 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:47.132919    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fg4gh_16129530-3326-4f01-8f22-b9ea97538fd9/kube-rbac-proxy/0.log"
Sep 19 11:16:47.153536 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:47.153508    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fg4gh_16129530-3326-4f01-8f22-b9ea97538fd9/exporter/0.log"
Sep 19 11:16:47.173843 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:47.173813    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fg4gh_16129530-3326-4f01-8f22-b9ea97538fd9/extractor/0.log"
Sep 19 11:16:48.186568 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:48.186543    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4wp77/perf-node-gather-daemonset-969qh"
Sep 19 11:16:49.562010 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:49.561961    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wwqqp_798963fa-d2b8-45c0-836e-4a8eb892c62e/manager/0.log"
Sep 19 11:16:50.065670 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:50.065630    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6qkpw_f2ab3b71-ec29-4b92-8165-2fd479aba8ca/manager/0.log"
Sep 19 11:16:50.292826 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:50.292786    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-h9wxr_04c58eb8-f14d-4b7a-8fc5-433a2c13cfbc/manager/0.log"
Sep 19 11:16:50.450206 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:50.450174    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt_43a85296-9b77-4c62-b5a5-ab4edc8e3e67/manager/0.log"
Sep 19 11:16:50.466484 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:50.466456    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d7b84ccc8-qkdgt_43a85296-9b77-4c62-b5a5-ab4edc8e3e67/copy-manifests/0.log"
Sep 19 11:16:52.090861 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:52.090740    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-v5jll_41988144-a03c-44a4-8729-d4b7d61116a8/openshift-lws-operator/0.log"
Sep 19 11:16:53.365586 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:53.365558    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d97bd96f8-tghnl_688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa/kube-auth-proxy/0.log"
Sep 19 11:16:55.442748 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:55.442719    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d97bd96f8-tghnl_688d0c55-0ec7-4ed4-914e-53ba8f4ec0aa/kube-auth-proxy/0.log"
Sep 19 11:16:56.285015 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:56.284979    2580 scope.go:117] "RemoveContainer" containerID="3d6c63fea616a324a361b2058237bcfa108750b69a696784ea59085139da4e0d"
Sep 19 11:16:58.833990 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:58.833959    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9jvzl_b8d90df9-5745-4b48-bd08-b5cdb66c5c9d/kube-multus-additional-cni-plugins/0.log"
Sep 19 11:16:58.853276 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:58.853247    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9jvzl_b8d90df9-5745-4b48-bd08-b5cdb66c5c9d/egress-router-binary-copy/0.log"
Sep 19 11:16:58.872769 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:58.872732    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9jvzl_b8d90df9-5745-4b48-bd08-b5cdb66c5c9d/cni-plugins/0.log"
Sep 19 11:16:58.891553 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:58.891512    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9jvzl_b8d90df9-5745-4b48-bd08-b5cdb66c5c9d/bond-cni-plugin/0.log"
Sep 19 11:16:58.909874 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:58.909848    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9jvzl_b8d90df9-5745-4b48-bd08-b5cdb66c5c9d/routeoverride-cni/0.log"
Sep 19 11:16:58.928176 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:58.928149    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9jvzl_b8d90df9-5745-4b48-bd08-b5cdb66c5c9d/whereabouts-cni-bincopy/0.log"
Sep 19 11:16:58.947701 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:58.947671    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9jvzl_b8d90df9-5745-4b48-bd08-b5cdb66c5c9d/whereabouts-cni/0.log"
Sep 19 11:16:59.319015 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:59.318972    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bj85t_238d9df9-4e46-4f9f-bcae-6d7e4a7913d6/kube-multus/0.log"
Sep 19 11:16:59.517803 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:59.517768    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fhc4f_162318c8-6ce5-4dd0-9f6c-6c61d1858816/network-metrics-daemon/0.log"
Sep 19 11:16:59.532267 ip-10-0-129-220 kubenswrapper[2580]: I0919 11:16:59.532247    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fhc4f_162318c8-6ce5-4dd0-9f6c-6c61d1858816/kube-rbac-proxy/0.log"