Sep 09 20:14:32.553958 ip-10-0-143-241 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:14:32.553971 ip-10-0-143-241 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:14:32.553979 ip-10-0-143-241 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:14:32.554189 ip-10-0-143-241 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:14:42.786758 ip-10-0-143-241 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 20:14:42.786775 ip-10-0-143-241 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5d7fda5cb23b4e578ca2217df94f1df4 --
Sep 09 20:16:32.636885 ip-10-0-143-241 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:16:33.051788 ip-10-0-143-241 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:16:33.051788 ip-10-0-143-241 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:16:33.051788 ip-10-0-143-241 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:16:33.051788 ip-10-0-143-241 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:16:33.051788 ip-10-0-143-241 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:16:33.053104 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.053007    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:16:33.056870 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056852    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:16:33.056870 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056869    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056876    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056881    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056884    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056887    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056892    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056896    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056899    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056902    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056905    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056908    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056911    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056913    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056916    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056918    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056921    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056924    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056927    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056930    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:16:33.056941 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056933    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056935    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056938    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056940    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056944    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056948    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056951    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056954    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056958    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056961    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056963    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056966    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056968    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056971    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056973    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056976    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056978    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056981    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056983    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:16:33.057392 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056986    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056989    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056991    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056994    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.056997    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057000    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057002    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057005    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057007    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057016    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057019    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057021    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057024    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057027    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057029    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057033    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057035    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057038    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057041    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057044    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:16:33.057887 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057046    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057049    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057052    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057054    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057057    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057060    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057062    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057065    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057067    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057070    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057073    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057075    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057078    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057080    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057083    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057085    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057089    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057092    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057095    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057097    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:16:33.058458 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057100    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057103    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057105    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057107    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057110    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057113    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057115    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057119    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057122    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057125    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057127    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057130    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057133    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057135    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057138    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057140    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057143    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057145    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057148    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057151    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:16:33.058963 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057889    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057897    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057900    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057903    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057905    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057910    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057914    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057917    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057920    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057924    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057927    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057931    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057935    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057938    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057941    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057944    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057947    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057949    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057952    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:16:33.059444 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057955    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057958    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057961    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057964    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057967    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057970    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057972    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057975    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057977    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057980    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057982    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057985    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057987    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057991    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057994    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.057997    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058000    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058002    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058005    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:16:33.059968 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058007    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058011    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058014    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058016    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058019    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058021    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058025    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058027    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058029    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058032    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058034    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058037    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058040    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058043    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058046    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058049    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058051    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058054    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058057    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058059    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:16:33.060438 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058062    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058065    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058067    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058070    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058072    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058075    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058079    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058081    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058084    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058086    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058089    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058091    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058094    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058097    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058099    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058102    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058104    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058107    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058109    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058112    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:16:33.060997 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058115    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058117    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058119    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058122    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058124    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058128    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058131    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058133    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058136    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058138    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058141    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058143    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058146    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058148    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058151    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058153    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058156    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058158    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058161    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058164    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:16:33.061473 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058166    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058606    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058613    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058616    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058620    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058623    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058626    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058629    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058633    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058636    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058639    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058643    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058646    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058648    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058651    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058654    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058657    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058659    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058662    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058665    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:16:33.061985 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058668    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058671    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058673    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058676    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058678    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058681    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058683    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058686    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058688    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058691    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058694    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058696    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058699    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058702    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058704    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058707    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058710    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058712    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058715    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058718    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:16:33.062477 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058720    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058723    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058725    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058728    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058732    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058734    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058737    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058740    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058743    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058746    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058749    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058751    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058753    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058756    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058759    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058761    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058764    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058766    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058770    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:16:33.062976 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058773    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058776    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058779    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058782    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058784    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058787    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058790    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058793    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058795    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058797    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058800    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058802    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058805    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058808    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058810    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058813    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058815    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058818    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058822    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058824    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:16:33.063445 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058827    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058830    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058832    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058835    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058838    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058840    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058843    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058845    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058848    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058850    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058853    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058855    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058859    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058862    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058865    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058867    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058870    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058873    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058876    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058878    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:16:33.063955 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.058881    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.058959    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059025    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059035    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059042    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059049    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059053    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059057    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059061    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059065    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059068    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059074    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059078    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059081    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059084    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059088    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059090    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059093    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059097    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059102    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059104    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059107    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059110    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059114    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:16:33.064461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059117    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059120    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059124    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059127    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059130    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059133    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059136    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059140    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059143    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059147    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059150    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059153    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059156    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059159    2563 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059162    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059166    2563 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059169    2563 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059172    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059175    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059179    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059183    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059186    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059190    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059194    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059197    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:16:33.065057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059200    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059202    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059205    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059210    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059213    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059216    2563 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059220    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059223    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059226    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059229    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059232    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059235    2563 flags.go:64] FLAG: --help="false"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059238    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059241    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059244    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059247    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059250    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059254    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059256    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059259    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059262    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059265    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059267    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059270    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:16:33.065692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059273    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059276    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059280    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059283    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059286    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059289    2563 flags.go:64] FLAG: --lock-file=""
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059292    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059295    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059298    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059304    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059307    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059311    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059314    2563 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059316    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059320    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059322    2563 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059325    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059330    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059333    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059337    2563 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059340    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059343    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059346    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059349    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059351    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:16:33.066319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059354    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059357    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059365    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059368    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059370    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059374    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059376    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059382    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059385    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059388    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059392    2563 flags.go:64] FLAG: --port="10250"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059395    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059399    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-062c617c2607b4d02"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059402    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059405    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059408    2563 flags.go:64] FLAG: --register-node="true"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059411    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059414    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059418    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059421    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059424    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059428    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059431    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059434    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059437    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:16:33.066957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059440    2563 flags.go:64] FLAG: --runonce="false"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059443    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059446    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059449    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059452    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059456    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059459    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059462    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059465    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059467    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059470    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059473    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059476    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059479    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059482    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059484    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059490    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059494    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059496    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059501    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059504    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059507    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059510    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059513    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059516    2563 flags.go:64] FLAG: --v="2"
Sep 09 20:16:33.067554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059521    2563 flags.go:64] FLAG: --version="false"
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059525    2563 flags.go:64] FLAG: --vmodule=""
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059529    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.059532    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059661    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059665    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059669    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059672    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059675    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059678    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059680    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059683    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059686    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059688    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059691    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059694    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059697    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059699    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059703    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:16:33.068223 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059708    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059711    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059714    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059718    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059721    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059725    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059728    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059731    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059734    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059738    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059741    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059743    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059746    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059751    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059755    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059758    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059761    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059764    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059767    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059770    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:16:33.068710 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059772    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059775    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059778    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059781    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059784    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059787    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059789    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059792    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059795    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059797    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059800    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059802    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059805    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059808    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059810    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059813    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059815    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059819    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059821    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059824    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:16:33.069207 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059827    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059829    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059832    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059835    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059837    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059841    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059843    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059846    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059849    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059852    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059854    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059857    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059859    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059862    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059864    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059867    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059870    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059872    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059874    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:16:33.069756 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059877    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059880    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059883    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059885    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059887    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059890    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059892    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059895    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059897    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059900    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059903    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059906    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059909    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059911    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059928    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059932    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059937    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059940    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059944    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059947    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:16:33.070241 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059949    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059952    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059955    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059958    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.059960    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.060756    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.068044    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.068065    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.069013    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.069026    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069105    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069111    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069114    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069117    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069120    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:16:33.070744 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069123    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069126    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069129    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069132    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069134    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069137    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069140    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069142    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069145    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069147    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069150    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069153    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069155    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069158    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069162    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069166    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069169    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069172    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069174    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:16:33.071126 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069177    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069180    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069183    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069185    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069188    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069199    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069202    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069205    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069207    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069211    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069215    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069218    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069221    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069223    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069226    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069230    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069234    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069237    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069240    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069243    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:16:33.071646 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069245    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069248    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069251    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069254    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069256    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069259    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069262    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069265    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069267    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069270    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069272    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069275    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069278    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069280    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069283    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069285    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069288    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069292    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069300    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069303    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:16:33.072131 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069306    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069308    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069311    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069313    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069316    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069318    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069321    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069324    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069326    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069329    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069331    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069334    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069336    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069339    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069341    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069344    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069346    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069349    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069351    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069354    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:16:33.072638 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069356    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069359    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069362    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069365    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069367    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069370    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069372    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069375    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069378    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069381    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069384    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069392    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069395    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069398    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069400    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:16:33.073130 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.069405    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069561    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069566    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069594    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069599    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069602    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069605    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069608    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069611    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069615    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069618    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069621    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069624    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069627    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069630    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069633    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069636    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069639    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069642    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:16:33.073501 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069645    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069647    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069650    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069652    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069655    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069657    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069660    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069663    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069667    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069670    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069679    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069682    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069685    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069687    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069690    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069692    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069695    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069698    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069700    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069703    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:16:33.074118 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069706    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069708    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069711    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069714    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069716    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069719    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069721    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069724    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069726    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069729    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069731    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069733    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069736    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069739    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069741    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069744    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069746    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069749    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069751    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:16:33.074605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069755    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069757    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069760    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069763    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069772    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069775    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069778    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069780    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069783    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069785    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069788    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069790    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069793    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069795    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069798    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069800    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069803    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069805    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069815    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069818    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:16:33.075060 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069821    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069823    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069826    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069828    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069831    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069834    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069836    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069839    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069841    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069844    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069846    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069849    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069852    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069855    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069858    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069860    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069863    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069871    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069874    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069876    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:16:33.075536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069879    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:16:33.076043 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:33.069881    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:16:33.076043 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.069886    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:16:33.076043 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.070076    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:16:33.076043 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.072080    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:16:33.076043 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.072979    2563 server.go:1019] "Starting client certificate rotation"
Sep 09 20:16:33.076043 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.073084    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:16:33.076043 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.073128    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:16:33.076043 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.076029    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:16:33.083824 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.083792    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:16:33.095160 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.095136    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:16:33.098044 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.098028    2563 log.go:25] "Validated CRI v1 image API"
Sep 09 20:16:33.098862 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.098841    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:16:33.101659 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.101638    2563 fs.go:135] Filesystem UUIDs: map[4b8c1338-1ea9-477e-b3ae-8edaf094e618:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e53f94e9-f50a-405a-bb6b-9855c10995de:/dev/nvme0n1p4]
Sep 09 20:16:33.101736 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.101657    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:16:33.103632 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.103612    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:16:33.107120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.107005    2563 manager.go:216] Machine: {Timestamp:2026-09-09 20:16:33.105604974 +0000 UTC m=+0.344707264 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098754 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec207e4c95a4136db313142a0f17c34e SystemUUID:ec207e4c-95a4-136d-b313-142a0f17c34e BootID:5d7fda5c-b23b-4e57-8ca2-217df94f1df4 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:12:ed:f9:e1:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:12:ed:f9:e1:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:e1:7a:61:d5:c5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:16:33.107120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.107119    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:16:33.107235 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.107202    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:16:33.110218 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.110182    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:16:33.110365 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.110220    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-241.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:16:33.110940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.110929    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:16:33.110977 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.110943    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:16:33.110977 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.110950    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:16:33.111631 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.111621    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:16:33.112872 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.112862    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:16:33.116733 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.116721    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:16:33.116893 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.116884    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:16:33.116934 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.116926    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:16:33.116966 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.116941    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:16:33.116966 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.116961    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:16:33.117019 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.116976    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:16:33.117019 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.116989    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:16:33.117485 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.117467    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:16:33.117590 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.117515    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:16:33.120241 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.120222    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:16:33.122340 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.122325    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:16:33.122451 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.122355    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:16:33.122507 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.122497    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:16:33.122537 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.122512    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:16:33.122537 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.122519    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:16:33.122537 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.122524    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:16:33.122537 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.122532    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:16:33.122537 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.122537    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:16:33.123195 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.123186    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:16:33.123195 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.123195    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:16:33.123250 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.123206    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:16:33.123250 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.123211    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:16:33.123250 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.123220    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:16:33.123250 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.123228    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:16:33.124098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.124086    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:16:33.124132 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.124122    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:16:33.126054 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.126034    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-241.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:16:33.126677 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.126656    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:16:33.126740 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.126656    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:16:33.128068 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.128055    2563 server.go:1268] "Started kubelet"
Sep 09 20:16:33.128166 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.128138    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:16:33.131985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.128158    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:16:33.131985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.131687    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bd9lt"
Sep 09 20:16:33.131647 ip-10-0-143-241 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:16:33.132531 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.132318    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:16:33.132967 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.132742    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:16:33.135359 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.135338    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:16:33.137135 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.137115    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bd9lt"
Sep 09 20:16:33.137378 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.137353    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:16:33.138212 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.138197    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:16:33.138742 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.137691    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-241.ec2.internal.18d3c05f6fe6420a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-241.ec2.internal,UID:ip-10-0-143-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-241.ec2.internal,},FirstTimestamp:2026-09-09 20:16:33.128006154 +0000 UTC m=+0.367108444,LastTimestamp:2026-09-09 20:16:33.128006154 +0000 UTC m=+0.367108444,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-241.ec2.internal,}"
Sep 09 20:16:33.138871 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.138751    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:16:33.138989 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.138971    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:33.139553 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.139529    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:16:33.139553 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.139532    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:16:33.139688 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.139558    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:16:33.140404 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.140385    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:16:33.140404 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.140400    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:16:33.140630 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.140615    2563 factory.go:153] Registering CRI-O factory
Sep 09 20:16:33.140630 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.140630    2563 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:16:33.140739 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.140688    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:16:33.140739 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.140696    2563 factory.go:55] Registering systemd factory
Sep 09 20:16:33.140739 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.140701    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:16:33.140739 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.140718    2563 factory.go:103] Registering Raw factory
Sep 09 20:16:33.140739 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.140732    2563 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:16:33.141160 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.141125    2563 manager.go:349] Starting recovery of all containers
Sep 09 20:16:33.148170 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.147985    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:16:33.151028 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.151003    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-241.ec2.internal\" not found" node="ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.153612 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.153598    2563 manager.go:354] Recovery completed
Sep 09 20:16:33.155457 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.155436    2563 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 09 20:16:33.159711 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.159696    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:33.161557 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.161539    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:33.161641 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.161585    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:33.161641 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.161600    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:33.162551 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.162535    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:16:33.162551 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.162548    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:16:33.162718 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.162553    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:33.162718 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.162564    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:16:33.163295 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.163279    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:33.163361 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.163307    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:33.163361 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.163324    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:33.164559 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.164543    2563 policy_none.go:49] "None policy: Start"
Sep 09 20:16:33.164559 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.164560    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:16:33.164677 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.164583    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:16:33.166451 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.166439    2563 policy_none.go:47] "Start"
Sep 09 20:16:33.220907 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.201173    2563 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:16:33.220907 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.201212    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:16:33.220907 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.201225    2563 server.go:87] "Starting device plugin registration server"
Sep 09 20:16:33.220907 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.201599    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:16:33.220907 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.201614    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:16:33.220907 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.201713    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:16:33.220907 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.201829    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:16:33.220907 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.201840    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:16:33.220907 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.202492    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:16:33.220907 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.202545    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:33.301950 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.301847    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:33.302922 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.302903    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:33.303029 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.302938    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:33.303029 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.302951    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:33.303029 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.302985    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.305660 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.305635    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:16:33.306788 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.306771    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:16:33.306881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.306828    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:16:33.306881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.306853    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:16:33.306981 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.306964    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:16:33.309128 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.309105    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:16:33.309453 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.309436    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.309509 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.309463    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-241.ec2.internal\": node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:33.370421 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.370389    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:33.407495 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.407459    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"]
Sep 09 20:16:33.407647 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.407542    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:33.408652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.408636    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:33.408736 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.408667    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:33.408736 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.408678    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:33.410912 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.410899    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:33.411006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.410991    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.411040 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.411026    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:33.411710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.411695    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:33.411774 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.411723    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:33.411774 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.411742    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:33.412232 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.412218    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:33.412282 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.412245    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:33.412282 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.412255    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:33.413724 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.413711    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.413779 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.413735    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:33.414399 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.414384    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:33.414465 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.414411    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:33.414465 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.414422    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:33.439380 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.439357    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-241.ec2.internal\" not found" node="ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.441344 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.441320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20ada389393df0af1b24f49af160350a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-241.ec2.internal\" (UID: \"20ada389393df0af1b24f49af160350a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.441450 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.441353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a6999f592201b6868f277532693fdb1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"9a6999f592201b6868f277532693fdb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.441450 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.441381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a6999f592201b6868f277532693fdb1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"9a6999f592201b6868f277532693fdb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.443904 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.443887    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-241.ec2.internal\" not found" node="ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.471261 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.471228    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:33.542532 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.542492    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20ada389393df0af1b24f49af160350a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-241.ec2.internal\" (UID: \"20ada389393df0af1b24f49af160350a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.542532 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.542529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a6999f592201b6868f277532693fdb1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"9a6999f592201b6868f277532693fdb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.542765 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.542557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a6999f592201b6868f277532693fdb1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"9a6999f592201b6868f277532693fdb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.542765 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.542638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20ada389393df0af1b24f49af160350a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-241.ec2.internal\" (UID: \"20ada389393df0af1b24f49af160350a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.542765 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.542647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a6999f592201b6868f277532693fdb1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"9a6999f592201b6868f277532693fdb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.542765 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.542643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a6999f592201b6868f277532693fdb1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"9a6999f592201b6868f277532693fdb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.573533 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.573451    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:33.676135 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.676091    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:33.743453 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.743418    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.746887 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:33.746866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:33.777029 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.776988    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:33.879773 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.879682    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:33.980327 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:33.980291    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:34.074651 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:34.074610    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:16:34.075291 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:34.074787    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:16:34.075291 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:34.074791    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:16:34.081321 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:34.081292    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:34.142571 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:34.142485    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:11:34 +0000 UTC" deadline="2028-05-21 08:03:00.739340102 +0000 UTC"
Sep 09 20:16:34.142571 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:34.142533    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14867h46m26.596827219s"
Sep 09 20:16:34.181467 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:34.181421    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:34.282589 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:34.282541    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:34.299012 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:34.298988    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:16:34.334163 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:34.334132    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6999f592201b6868f277532693fdb1.slice/crio-726c234cf0604ba3ec81f31c3442382c1be799ef446ca8c0b8162dd66e1d011e WatchSource:0}: Error finding container 726c234cf0604ba3ec81f31c3442382c1be799ef446ca8c0b8162dd66e1d011e: Status 404 returned error can't find the container with id 726c234cf0604ba3ec81f31c3442382c1be799ef446ca8c0b8162dd66e1d011e
Sep 09 20:16:34.339245 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:34.339222    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:16:34.340098 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:34.340021    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ada389393df0af1b24f49af160350a.slice/crio-b9d2fd286c5dbe34430b6cfe34e89e50009dae01d559c55f29219f06009eb6b0 WatchSource:0}: Error finding container b9d2fd286c5dbe34430b6cfe34e89e50009dae01d559c55f29219f06009eb6b0: Status 404 returned error can't find the container with id b9d2fd286c5dbe34430b6cfe34e89e50009dae01d559c55f29219f06009eb6b0
Sep 09 20:16:34.384639 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:34.384598    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:34.484765 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:34.484678    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found"
Sep 09 20:16:34.549136 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:34.549101    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:16:34.640188 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:34.640148    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:34.651546 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:34.651525    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:16:34.652860 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:34.652847    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal"
Sep 09 20:16:34.664929 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:34.664910    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:16:35.070412 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.070378    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:16:35.118405 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.118181    2563 apiserver.go:52] "Watching apiserver"
Sep 09 20:16:35.127587 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.127558    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:16:35.128849 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.128817    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-lsr8k","kube-system/konnectivity-agent-jx5cd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t","openshift-cluster-node-tuning-operator/tuned-vdbpg","openshift-dns/node-resolver-zsfbn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal","openshift-multus/network-metrics-daemon-86bgz","openshift-network-diagnostics/network-check-target-pkqbv","kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal","openshift-image-registry/node-ca-t64d7","openshift-multus/multus-additional-cni-plugins-klsl6","openshift-multus/multus-c72np","openshift-network-operator/iptables-alerter-h4vr2","openshift-ovn-kubernetes/ovnkube-node-8hcfl"]
Sep 09 20:16:35.131748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.131725    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:35.131874 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.131815    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:16:35.133997 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.133971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h4vr2"
Sep 09 20:16:35.136323 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.136113    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.139287 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.139268    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.140901 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.140876    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:16:35.141010 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.140929    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:16:35.141135 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.141117    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:16:35.141213 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.141188    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bbdmt\""
Sep 09 20:16:35.141502 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.141418    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:16:35.141502 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.141477    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zsfbn"
Sep 09 20:16:35.142234 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.141933    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j84lm\""
Sep 09 20:16:35.142234 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.142152    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:16:35.142384 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.142370    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:16:35.142596 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.142567    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:16:35.143711 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.143692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.145306 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.145283    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:16:35.147459 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.146336    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:35.147459 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.146407    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:16:35.147616 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.147538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c9c2n\""
Sep 09 20:16:35.148494 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.148470    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:16:35.148807 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.148787    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:16:35.149262 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.149244    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:35.149339 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.149311    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:16:35.153019 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.152997    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j7rzk\""
Sep 09 20:16:35.153121 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.153038    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:16:35.153268 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.153250    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:16:35.153329 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.153311    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:16:35.153527 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.153510    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:16:35.153590 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.153579    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:16:35.153695 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.153513    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cd929\""
Sep 09 20:16:35.153908 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.153261    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:16:35.153908 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.153821    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:16:35.154051 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.153964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jx5cd"
Sep 09 20:16:35.156198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.155441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-sys\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.156198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.155943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-socket-dir\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.156198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.155975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.156198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.156198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrw9\" (UniqueName: \"kubernetes.io/projected/faac4705-5d14-4b37-8a0f-a03925b24e5c-kube-api-access-lhrw9\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.156198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-kubernetes\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.156198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-sysctl-d\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.156198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-lib-modules\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.156198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-tuned\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29247776-5cf6-420f-897b-18d0d9ae3a7a-iptables-alerter-script\") pod \"iptables-alerter-h4vr2\" (UID: \"29247776-5cf6-420f-897b-18d0d9ae3a7a\") " pod="openshift-network-operator/iptables-alerter-h4vr2"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29247776-5cf6-420f-897b-18d0d9ae3a7a-host-slash\") pod \"iptables-alerter-h4vr2\" (UID: \"29247776-5cf6-420f-897b-18d0d9ae3a7a\") " pod="openshift-network-operator/iptables-alerter-h4vr2"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-sysctl-conf\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-var-lib-kubelet\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1833449f-dfe7-4c13-ae92-562b409d1cf6-tmp-dir\") pod \"node-resolver-zsfbn\" (UID: \"1833449f-dfe7-4c13-ae92-562b409d1cf6\") " pod="openshift-dns/node-resolver-zsfbn"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfhsj\" (UniqueName: \"kubernetes.io/projected/1833449f-dfe7-4c13-ae92-562b409d1cf6-kube-api-access-wfhsj\") pod \"node-resolver-zsfbn\" (UID: \"1833449f-dfe7-4c13-ae92-562b409d1cf6\") " pod="openshift-dns/node-resolver-zsfbn"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zhr\" (UniqueName: \"kubernetes.io/projected/29247776-5cf6-420f-897b-18d0d9ae3a7a-kube-api-access-w2zhr\") pod \"iptables-alerter-h4vr2\" (UID: \"29247776-5cf6-420f-897b-18d0d9ae3a7a\") " pod="openshift-network-operator/iptables-alerter-h4vr2"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-registration-dir\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-systemd\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be868ed8-2f22-414e-91f2-5e511d69b5b4-tmp\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k5gx\" (UniqueName: \"kubernetes.io/projected/be868ed8-2f22-414e-91f2-5e511d69b5b4-kube-api-access-9k5gx\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1833449f-dfe7-4c13-ae92-562b409d1cf6-hosts-file\") pod \"node-resolver-zsfbn\" (UID: \"1833449f-dfe7-4c13-ae92-562b409d1cf6\") " pod="openshift-dns/node-resolver-zsfbn"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6s5c\" (UniqueName: \"kubernetes.io/projected/7377be15-1184-4239-a696-75be3f1d835d-kube-api-access-p6s5c\") pod \"network-metrics-daemon-86bgz\" (UID: \"7377be15-1184-4239-a696-75be3f1d835d\") " pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-sysconfig\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-host\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.156740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs\") pod \"network-metrics-daemon-86bgz\" (UID: \"7377be15-1184-4239-a696-75be3f1d835d\") " pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:35.157748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-sys-fs\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.157748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-device-dir\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.157748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156764    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t64d7"
Sep 09 20:16:35.157748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-modprobe-d\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.157748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.156844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-run\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.159527 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.159508    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.160562 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.160543    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vf5kz\""
Sep 09 20:16:35.166091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.161771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.166091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.160872    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:16:35.166091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.162861    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:16:35.166091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.162902    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qmxpv\""
Sep 09 20:16:35.166091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.163190    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:16:35.166091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.160548    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:16:35.166091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.164665    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:16:35.166091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.164937    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:16:35.166091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.164959    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:16:35.166091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.165414    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:16:35.166091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.165741    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:16:35.166640 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.166578    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kg69k\""
Sep 09 20:16:35.167185 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.167165    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8sz2c\""
Sep 09 20:16:35.168310 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.168287    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:16:35.168418 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.168296    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:16:35.175570 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.175544    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:16:35.230082 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.230054    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jjrrl"
Sep 09 20:16:35.241521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.241495    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:16:35.247452 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.247430    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jjrrl"
Sep 09 20:16:35.257510 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.257480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8696794c-360f-4af6-bc12-81b6c8f58ea5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.257656 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.257546    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73d9e6c4-c63a-458e-a181-def2c4405b45-agent-certs\") pod \"konnectivity-agent-jx5cd\" (UID: \"73d9e6c4-c63a-458e-a181-def2c4405b45\") " pod="kube-system/konnectivity-agent-jx5cd"
Sep 09 20:16:35.257656 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.257642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-multus-socket-dir-parent\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.257790 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.257694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-multus-conf-dir\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.257790 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.257742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.257790 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.257770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrw9\" (UniqueName: \"kubernetes.io/projected/faac4705-5d14-4b37-8a0f-a03925b24e5c-kube-api-access-lhrw9\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.257947 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.257796    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-kubernetes\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.257947 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.257823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-sysctl-d\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.257947 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.257860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm4rt\" (UniqueName: \"kubernetes.io/projected/8696794c-360f-4af6-bc12-81b6c8f58ea5-kube-api-access-qm4rt\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.258153 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-sysctl-d\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.258153 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-kubelet\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.258153 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.258153 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9af1130-d46a-415f-9086-c10df2dd7ad5-cni-binary-copy\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.258153 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.258397 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258177    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1833449f-dfe7-4c13-ae92-562b409d1cf6-tmp-dir\") pod \"node-resolver-zsfbn\" (UID: \"1833449f-dfe7-4c13-ae92-562b409d1cf6\") " pod="openshift-dns/node-resolver-zsfbn"
Sep 09 20:16:35.258397 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-cnibin\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.258397 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-kubernetes\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.258397 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-run-k8s-cni-cncf-io\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.258397 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-etc-kubernetes\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.258397 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258360    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be868ed8-2f22-414e-91f2-5e511d69b5b4-tmp\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.258397 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8696794c-360f-4af6-bc12-81b6c8f58ea5-cnibin\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-run-multus-certs\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-sysconfig\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-var-lib-openvswitch\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-etc-openvswitch\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-ovnkube-script-lib\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs\") pod \"network-metrics-daemon-86bgz\" (UID: \"7377be15-1184-4239-a696-75be3f1d835d\") " pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-sys-fs\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-sysconfig\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d65fd73a-56b3-4e21-988d-cd2d26bb8847-kubelet-config\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.258677    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8696794c-360f-4af6-bc12-81b6c8f58ea5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-sys-fs\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.258762    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs podName:7377be15-1184-4239-a696-75be3f1d835d nodeName:}" failed. No retries permitted until 2026-09-09 20:16:35.75874052 +0000 UTC m=+2.997842800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs") pod "network-metrics-daemon-86bgz" (UID: "7377be15-1184-4239-a696-75be3f1d835d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-run-ovn\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-system-cni-dir\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258850    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:16:35.258914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-var-lib-kubelet\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-hostroot\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-device-dir\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.258998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gdnp\" (UniqueName: \"kubernetes.io/projected/36251d92-c8c9-450d-ac2c-103c4d5d38e6-kube-api-access-8gdnp\") pod \"node-ca-t64d7\" (UID: \"36251d92-c8c9-450d-ac2c-103c4d5d38e6\") " pod="openshift-image-registry/node-ca-t64d7"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-cni-netd\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-device-dir\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrr8x\" (UniqueName: \"kubernetes.io/projected/a9af1130-d46a-415f-9086-c10df2dd7ad5-kube-api-access-xrr8x\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-run\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-run\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-node-log\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-ovnkube-config\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1833449f-dfe7-4c13-ae92-562b409d1cf6-tmp-dir\") pod \"node-resolver-zsfbn\" (UID: \"1833449f-dfe7-4c13-ae92-562b409d1cf6\") " pod="openshift-dns/node-resolver-zsfbn"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73d9e6c4-c63a-458e-a181-def2c4405b45-konnectivity-ca\") pod \"konnectivity-agent-jx5cd\" (UID: \"73d9e6c4-c63a-458e-a181-def2c4405b45\") " pod="kube-system/konnectivity-agent-jx5cd"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-socket-dir\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36251d92-c8c9-450d-ac2c-103c4d5d38e6-serviceca\") pod \"node-ca-t64d7\" (UID: \"36251d92-c8c9-450d-ac2c-103c4d5d38e6\") " pod="openshift-image-registry/node-ca-t64d7"
Sep 09 20:16:35.259794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfsrl\" (UniqueName: \"kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl\") pod \"network-check-target-pkqbv\" (UID: \"011ed84f-e0dd-495c-9eb3-fd0ad001c3d5\") " pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-run-netns\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-log-socket\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-socket-dir\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-lib-modules\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8696794c-360f-4af6-bc12-81b6c8f58ea5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-env-overrides\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9af1130-d46a-415f-9086-c10df2dd7ad5-multus-daemon-config\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-tuned\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29247776-5cf6-420f-897b-18d0d9ae3a7a-iptables-alerter-script\") pod \"iptables-alerter-h4vr2\" (UID: \"29247776-5cf6-420f-897b-18d0d9ae3a7a\") " pod="openshift-network-operator/iptables-alerter-h4vr2"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29247776-5cf6-420f-897b-18d0d9ae3a7a-host-slash\") pod \"iptables-alerter-h4vr2\" (UID: \"29247776-5cf6-420f-897b-18d0d9ae3a7a\") " pod="openshift-network-operator/iptables-alerter-h4vr2"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-sysctl-conf\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-lib-modules\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-var-lib-kubelet\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfhsj\" (UniqueName: \"kubernetes.io/projected/1833449f-dfe7-4c13-ae92-562b409d1cf6-kube-api-access-wfhsj\") pod \"node-resolver-zsfbn\" (UID: \"1833449f-dfe7-4c13-ae92-562b409d1cf6\") " pod="openshift-dns/node-resolver-zsfbn"
Sep 09 20:16:35.260637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29247776-5cf6-420f-897b-18d0d9ae3a7a-host-slash\") pod \"iptables-alerter-h4vr2\" (UID: \"29247776-5cf6-420f-897b-18d0d9ae3a7a\") " pod="openshift-network-operator/iptables-alerter-h4vr2"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d65fd73a-56b3-4e21-988d-cd2d26bb8847-dbus\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zhr\" (UniqueName: \"kubernetes.io/projected/29247776-5cf6-420f-897b-18d0d9ae3a7a-kube-api-access-w2zhr\") pod \"iptables-alerter-h4vr2\" (UID: \"29247776-5cf6-420f-897b-18d0d9ae3a7a\") " pod="openshift-network-operator/iptables-alerter-h4vr2"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-var-lib-kubelet\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.259972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-registration-dir\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-systemd\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5gx\" (UniqueName: \"kubernetes.io/projected/be868ed8-2f22-414e-91f2-5e511d69b5b4-kube-api-access-9k5gx\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1833449f-dfe7-4c13-ae92-562b409d1cf6-hosts-file\") pod \"node-resolver-zsfbn\" (UID: \"1833449f-dfe7-4c13-ae92-562b409d1cf6\") " pod="openshift-dns/node-resolver-zsfbn"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-sysctl-conf\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36251d92-c8c9-450d-ac2c-103c4d5d38e6-host\") pod \"node-ca-t64d7\" (UID: \"36251d92-c8c9-450d-ac2c-103c4d5d38e6\") " pod="openshift-image-registry/node-ca-t64d7"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1833449f-dfe7-4c13-ae92-562b409d1cf6-hosts-file\") pod \"node-resolver-zsfbn\" (UID: \"1833449f-dfe7-4c13-ae92-562b409d1cf6\") " pod="openshift-dns/node-resolver-zsfbn"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8696794c-360f-4af6-bc12-81b6c8f58ea5-cni-binary-copy\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/faac4705-5d14-4b37-8a0f-a03925b24e5c-registration-dir\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6s5c\" (UniqueName: \"kubernetes.io/projected/7377be15-1184-4239-a696-75be3f1d835d-kube-api-access-p6s5c\") pod \"network-metrics-daemon-86bgz\" (UID: \"7377be15-1184-4239-a696-75be3f1d835d\") " pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-run-netns\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-systemd\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-run-systemd\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.261521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-host\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-run-openvswitch\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-multus-cni-dir\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29247776-5cf6-420f-897b-18d0d9ae3a7a-iptables-alerter-script\") pod \"iptables-alerter-h4vr2\" (UID: \"29247776-5cf6-420f-897b-18d0d9ae3a7a\") " pod="openshift-network-operator/iptables-alerter-h4vr2"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-host\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-modprobe-d\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-slash\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-ovn-node-metrics-cert\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p58vk\" (UniqueName: \"kubernetes.io/projected/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-kube-api-access-p58vk\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-sys\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-modprobe-d\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-systemd-units\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-cni-bin\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-var-lib-cni-bin\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be868ed8-2f22-414e-91f2-5e511d69b5b4-sys\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-var-lib-cni-multus\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.262398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.260732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8696794c-360f-4af6-bc12-81b6c8f58ea5-system-cni-dir\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.264232 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.264208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be868ed8-2f22-414e-91f2-5e511d69b5b4-etc-tuned\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.265583 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.265563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be868ed8-2f22-414e-91f2-5e511d69b5b4-tmp\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.283168 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.283097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrw9\" (UniqueName: \"kubernetes.io/projected/faac4705-5d14-4b37-8a0f-a03925b24e5c-kube-api-access-lhrw9\") pod \"aws-ebs-csi-driver-node-v4l6t\" (UID: \"faac4705-5d14-4b37-8a0f-a03925b24e5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.283952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.283923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfhsj\" (UniqueName: \"kubernetes.io/projected/1833449f-dfe7-4c13-ae92-562b409d1cf6-kube-api-access-wfhsj\") pod \"node-resolver-zsfbn\" (UID: \"1833449f-dfe7-4c13-ae92-562b409d1cf6\") " pod="openshift-dns/node-resolver-zsfbn"
Sep 09 20:16:35.286915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.286867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k5gx\" (UniqueName: \"kubernetes.io/projected/be868ed8-2f22-414e-91f2-5e511d69b5b4-kube-api-access-9k5gx\") pod \"tuned-vdbpg\" (UID: \"be868ed8-2f22-414e-91f2-5e511d69b5b4\") " pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.302350 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.302293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6s5c\" (UniqueName: \"kubernetes.io/projected/7377be15-1184-4239-a696-75be3f1d835d-kube-api-access-p6s5c\") pod \"network-metrics-daemon-86bgz\" (UID: \"7377be15-1184-4239-a696-75be3f1d835d\") " pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:35.302519 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.302494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zhr\" (UniqueName: \"kubernetes.io/projected/29247776-5cf6-420f-897b-18d0d9ae3a7a-kube-api-access-w2zhr\") pod \"iptables-alerter-h4vr2\" (UID: \"29247776-5cf6-420f-897b-18d0d9ae3a7a\") " pod="openshift-network-operator/iptables-alerter-h4vr2"
Sep 09 20:16:35.312124 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.312067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" event={"ID":"20ada389393df0af1b24f49af160350a","Type":"ContainerStarted","Data":"b9d2fd286c5dbe34430b6cfe34e89e50009dae01d559c55f29219f06009eb6b0"}
Sep 09 20:16:35.313858 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.313375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" event={"ID":"9a6999f592201b6868f277532693fdb1","Type":"ContainerStarted","Data":"726c234cf0604ba3ec81f31c3442382c1be799ef446ca8c0b8162dd66e1d011e"}
Sep 09 20:16:35.361945 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.361855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-slash\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.361945 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.361896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-ovn-node-metrics-cert\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.361945 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.361926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p58vk\" (UniqueName: \"kubernetes.io/projected/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-kube-api-access-p58vk\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.361951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-systemd-units\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.361971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-slash\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.361976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-cni-bin\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-cni-bin\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-var-lib-cni-bin\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-var-lib-cni-multus\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8696794c-360f-4af6-bc12-81b6c8f58ea5-system-cni-dir\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8696794c-360f-4af6-bc12-81b6c8f58ea5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73d9e6c4-c63a-458e-a181-def2c4405b45-agent-certs\") pod \"konnectivity-agent-jx5cd\" (UID: \"73d9e6c4-c63a-458e-a181-def2c4405b45\") " pod="kube-system/konnectivity-agent-jx5cd"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-multus-socket-dir-parent\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-multus-conf-dir\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm4rt\" (UniqueName: \"kubernetes.io/projected/8696794c-360f-4af6-bc12-81b6c8f58ea5-kube-api-access-qm4rt\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-kubelet\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.362297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9af1130-d46a-415f-9086-c10df2dd7ad5-cni-binary-copy\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-cnibin\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-run-k8s-cni-cncf-io\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-etc-kubernetes\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8696794c-360f-4af6-bc12-81b6c8f58ea5-cnibin\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-run-multus-certs\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-var-lib-openvswitch\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-etc-openvswitch\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-etc-openvswitch\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-kubelet\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-var-lib-cni-bin\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-var-lib-cni-multus\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.362926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.362919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8696794c-360f-4af6-bc12-81b6c8f58ea5-system-cni-dir\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.363579 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8696794c-360f-4af6-bc12-81b6c8f58ea5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.363579 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9af1130-d46a-415f-9086-c10df2dd7ad5-cni-binary-copy\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.363579 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-cnibin\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.363579 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-run-k8s-cni-cncf-io\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.363579 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-etc-kubernetes\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.363851 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8696794c-360f-4af6-bc12-81b6c8f58ea5-cnibin\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.363851 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-run-multus-certs\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.363851 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-var-lib-openvswitch\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.363851 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-systemd-units\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.363851 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-multus-conf-dir\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.363851 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-ovnkube-script-lib\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d65fd73a-56b3-4e21-988d-cd2d26bb8847-kubelet-config\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8696794c-360f-4af6-bc12-81b6c8f58ea5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-run-ovn\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-system-cni-dir\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.363984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d65fd73a-56b3-4e21-988d-cd2d26bb8847-kubelet-config\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-var-lib-kubelet\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-hostroot\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-run-ovn\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gdnp\" (UniqueName: \"kubernetes.io/projected/36251d92-c8c9-450d-ac2c-103c4d5d38e6-kube-api-access-8gdnp\") pod \"node-ca-t64d7\" (UID: \"36251d92-c8c9-450d-ac2c-103c4d5d38e6\") " pod="openshift-image-registry/node-ca-t64d7"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-cni-netd\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrr8x\" (UniqueName: \"kubernetes.io/projected/a9af1130-d46a-415f-9086-c10df2dd7ad5-kube-api-access-xrr8x\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-node-log\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.364203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-ovnkube-config\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73d9e6c4-c63a-458e-a181-def2c4405b45-konnectivity-ca\") pod \"konnectivity-agent-jx5cd\" (UID: \"73d9e6c4-c63a-458e-a181-def2c4405b45\") " pod="kube-system/konnectivity-agent-jx5cd"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36251d92-c8c9-450d-ac2c-103c4d5d38e6-serviceca\") pod \"node-ca-t64d7\" (UID: \"36251d92-c8c9-450d-ac2c-103c4d5d38e6\") " pod="openshift-image-registry/node-ca-t64d7"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfsrl\" (UniqueName: \"kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl\") pod \"network-check-target-pkqbv\" (UID: \"011ed84f-e0dd-495c-9eb3-fd0ad001c3d5\") " pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-run-netns\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-log-socket\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8696794c-360f-4af6-bc12-81b6c8f58ea5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-env-overrides\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9af1130-d46a-415f-9086-c10df2dd7ad5-multus-daemon-config\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d65fd73a-56b3-4e21-988d-cd2d26bb8847-dbus\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36251d92-c8c9-450d-ac2c-103c4d5d38e6-host\") pod \"node-ca-t64d7\" (UID: \"36251d92-c8c9-450d-ac2c-103c4d5d38e6\") " pod="openshift-image-registry/node-ca-t64d7"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8696794c-360f-4af6-bc12-81b6c8f58ea5-cni-binary-copy\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-run-netns\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-run-systemd\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8696794c-360f-4af6-bc12-81b6c8f58ea5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-run-openvswitch\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.364915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-run-openvswitch\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.365607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-multus-cni-dir\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.365607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-multus-cni-dir\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.365607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-ovnkube-config\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.365607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365085    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-system-cni-dir\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.365607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-var-lib-kubelet\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.365607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-hostroot\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.365607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73d9e6c4-c63a-458e-a181-def2c4405b45-konnectivity-ca\") pod \"konnectivity-agent-jx5cd\" (UID: \"73d9e6c4-c63a-458e-a181-def2c4405b45\") " pod="kube-system/konnectivity-agent-jx5cd"
Sep 09 20:16:35.365607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365275    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9af1130-d46a-415f-9086-c10df2dd7ad5-multus-daemon-config\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.365607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-cni-netd\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.365607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-node-log\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.365607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.365607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-log-socket\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.366197 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36251d92-c8c9-450d-ac2c-103c4d5d38e6-serviceca\") pod \"node-ca-t64d7\" (UID: \"36251d92-c8c9-450d-ac2c-103c4d5d38e6\") " pod="openshift-image-registry/node-ca-t64d7"
Sep 09 20:16:35.366197 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-host-run-netns\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.366197 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-run-systemd\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.366197 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-host-run-netns\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.366197 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.365971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d65fd73a-56b3-4e21-988d-cd2d26bb8847-dbus\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:35.366197 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.365245    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:35.366197 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.366016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36251d92-c8c9-450d-ac2c-103c4d5d38e6-host\") pod \"node-ca-t64d7\" (UID: \"36251d92-c8c9-450d-ac2c-103c4d5d38e6\") " pod="openshift-image-registry/node-ca-t64d7"
Sep 09 20:16:35.366197 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.366089    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret podName:d65fd73a-56b3-4e21-988d-cd2d26bb8847 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:35.866050964 +0000 UTC m=+3.105153263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret") pod "global-pull-secret-syncer-lsr8k" (UID: "d65fd73a-56b3-4e21-988d-cd2d26bb8847") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:35.366197 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.366182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8696794c-360f-4af6-bc12-81b6c8f58ea5-cni-binary-copy\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.366674 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.364505    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-ovnkube-script-lib\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.366674 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.366206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-ovn-node-metrics-cert\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.366674 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.366313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-env-overrides\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.366674 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.366492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8696794c-360f-4af6-bc12-81b6c8f58ea5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.366674 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.366547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9af1130-d46a-415f-9086-c10df2dd7ad5-multus-socket-dir-parent\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.368514 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.368487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73d9e6c4-c63a-458e-a181-def2c4405b45-agent-certs\") pod \"konnectivity-agent-jx5cd\" (UID: \"73d9e6c4-c63a-458e-a181-def2c4405b45\") " pod="kube-system/konnectivity-agent-jx5cd"
Sep 09 20:16:35.375710 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.375688    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:16:35.375846 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.375715    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:16:35.375846 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.375730    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfsrl for pod openshift-network-diagnostics/network-check-target-pkqbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:35.375846 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.375820    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl podName:011ed84f-e0dd-495c-9eb3-fd0ad001c3d5 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:35.87579934 +0000 UTC m=+3.114901619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cfsrl" (UniqueName: "kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl") pod "network-check-target-pkqbv" (UID: "011ed84f-e0dd-495c-9eb3-fd0ad001c3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:35.377091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.377062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p58vk\" (UniqueName: \"kubernetes.io/projected/d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277-kube-api-access-p58vk\") pod \"ovnkube-node-8hcfl\" (UID: \"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.378685 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.378644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm4rt\" (UniqueName: \"kubernetes.io/projected/8696794c-360f-4af6-bc12-81b6c8f58ea5-kube-api-access-qm4rt\") pod \"multus-additional-cni-plugins-klsl6\" (UID: \"8696794c-360f-4af6-bc12-81b6c8f58ea5\") " pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.383206 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.383181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gdnp\" (UniqueName: \"kubernetes.io/projected/36251d92-c8c9-450d-ac2c-103c4d5d38e6-kube-api-access-8gdnp\") pod \"node-ca-t64d7\" (UID: \"36251d92-c8c9-450d-ac2c-103c4d5d38e6\") " pod="openshift-image-registry/node-ca-t64d7"
Sep 09 20:16:35.383443 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.383420    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrr8x\" (UniqueName: \"kubernetes.io/projected/a9af1130-d46a-415f-9086-c10df2dd7ad5-kube-api-access-xrr8x\") pod \"multus-c72np\" (UID: \"a9af1130-d46a-415f-9086-c10df2dd7ad5\") " pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.453802 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.453769    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h4vr2"
Sep 09 20:16:35.464615 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.464567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t"
Sep 09 20:16:35.482803 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.482774    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vdbpg"
Sep 09 20:16:35.490844 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.490818    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zsfbn"
Sep 09 20:16:35.499714 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.499691    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:16:35.510272 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.510248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jx5cd"
Sep 09 20:16:35.518840 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.518815    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t64d7"
Sep 09 20:16:35.527853 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.527829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-klsl6"
Sep 09 20:16:35.535101 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.535068    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c72np"
Sep 09 20:16:35.614491 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.614398    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:16:35.767170 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.767126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs\") pod \"network-metrics-daemon-86bgz\" (UID: \"7377be15-1184-4239-a696-75be3f1d835d\") " pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:35.767349 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.767305    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:35.767416 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.767384    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs podName:7377be15-1184-4239-a696-75be3f1d835d nodeName:}" failed. No retries permitted until 2026-09-09 20:16:36.767360243 +0000 UTC m=+4.006462539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs") pod "network-metrics-daemon-86bgz" (UID: "7377be15-1184-4239-a696-75be3f1d835d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:35.868661 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.868557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:35.868827 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.868742    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:35.868827 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.868820    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret podName:d65fd73a-56b3-4e21-988d-cd2d26bb8847 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:36.868803344 +0000 UTC m=+4.107905631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret") pod "global-pull-secret-syncer-lsr8k" (UID: "d65fd73a-56b3-4e21-988d-cd2d26bb8847") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:35.969315 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:35.969278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfsrl\" (UniqueName: \"kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl\") pod \"network-check-target-pkqbv\" (UID: \"011ed84f-e0dd-495c-9eb3-fd0ad001c3d5\") " pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:35.969507 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.969421    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:16:35.969507 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.969439    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:16:35.969507 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.969449    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfsrl for pod openshift-network-diagnostics/network-check-target-pkqbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:35.969716 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:35.969510    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl podName:011ed84f-e0dd-495c-9eb3-fd0ad001c3d5 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:36.969493059 +0000 UTC m=+4.208595341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfsrl" (UniqueName: "kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl") pod "network-check-target-pkqbv" (UID: "011ed84f-e0dd-495c-9eb3-fd0ad001c3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:36.251021 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:36.250932    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:11:36 +0000 UTC" deadline="2028-05-12 15:58:48.216774867 +0000 UTC"
Sep 09 20:16:36.251021 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:36.250970    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14659h42m11.965810707s"
Sep 09 20:16:36.307667 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:36.307635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:36.307851 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:36.307770    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:16:36.547986 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:36.547949    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9af1130_d46a_415f_9086_c10df2dd7ad5.slice/crio-50df7d8e7a4e411392d4d5c645f38daaf76f79c04abd0642c0f94420faacd507 WatchSource:0}: Error finding container 50df7d8e7a4e411392d4d5c645f38daaf76f79c04abd0642c0f94420faacd507: Status 404 returned error can't find the container with id 50df7d8e7a4e411392d4d5c645f38daaf76f79c04abd0642c0f94420faacd507
Sep 09 20:16:36.551224 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:36.550965    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1833449f_dfe7_4c13_ae92_562b409d1cf6.slice/crio-d44db6a7a94e75ee89623c430aaa415209bb5faab57e2360809c27fe5be39345 WatchSource:0}: Error finding container d44db6a7a94e75ee89623c430aaa415209bb5faab57e2360809c27fe5be39345: Status 404 returned error can't find the container with id d44db6a7a94e75ee89623c430aaa415209bb5faab57e2360809c27fe5be39345
Sep 09 20:16:36.554048 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:36.554011    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe868ed8_2f22_414e_91f2_5e511d69b5b4.slice/crio-ee44a9778353f8ec289d5b2c35a92ff1de93e4579e50e4d2d1001d35471a40af WatchSource:0}: Error finding container ee44a9778353f8ec289d5b2c35a92ff1de93e4579e50e4d2d1001d35471a40af: Status 404 returned error can't find the container with id ee44a9778353f8ec289d5b2c35a92ff1de93e4579e50e4d2d1001d35471a40af
Sep 09 20:16:36.555030 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:16:36.554992    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29247776_5cf6_420f_897b_18d0d9ae3a7a.slice/crio-469255b218b5076391c84aaaea22cd9cf827169a014bf3036491ddac56d518d4 WatchSource:0}: Error finding container 469255b218b5076391c84aaaea22cd9cf827169a014bf3036491ddac56d518d4: Status 404 returned error can't find the container with id 469255b218b5076391c84aaaea22cd9cf827169a014bf3036491ddac56d518d4
Sep 09 20:16:36.776187 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:36.775998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs\") pod \"network-metrics-daemon-86bgz\" (UID: \"7377be15-1184-4239-a696-75be3f1d835d\") " pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:36.776283 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:36.776152    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:36.776345 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:36.776286    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs podName:7377be15-1184-4239-a696-75be3f1d835d nodeName:}" failed. No retries permitted until 2026-09-09 20:16:38.77627071 +0000 UTC m=+6.015372990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs") pod "network-metrics-daemon-86bgz" (UID: "7377be15-1184-4239-a696-75be3f1d835d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:36.876994 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:36.876951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:36.877158 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:36.877096    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:36.877158 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:36.877157    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret podName:d65fd73a-56b3-4e21-988d-cd2d26bb8847 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:38.877140103 +0000 UTC m=+6.116242386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret") pod "global-pull-secret-syncer-lsr8k" (UID: "d65fd73a-56b3-4e21-988d-cd2d26bb8847") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:36.977829 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:36.977789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfsrl\" (UniqueName: \"kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl\") pod \"network-check-target-pkqbv\" (UID: \"011ed84f-e0dd-495c-9eb3-fd0ad001c3d5\") " pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:36.978042 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:36.978017    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:16:36.978125 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:36.978050    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:16:36.978125 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:36.978066    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfsrl for pod openshift-network-diagnostics/network-check-target-pkqbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:36.978232 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:36.978140    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl podName:011ed84f-e0dd-495c-9eb3-fd0ad001c3d5 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:38.97811963 +0000 UTC m=+6.217221934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfsrl" (UniqueName: "kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl") pod "network-check-target-pkqbv" (UID: "011ed84f-e0dd-495c-9eb3-fd0ad001c3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:37.254089 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.253936    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:11:36 +0000 UTC" deadline="2028-03-12 08:44:56.279298404 +0000 UTC"
Sep 09 20:16:37.254089 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.254005    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13188h28m19.025298014s"
Sep 09 20:16:37.309887 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.308813    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:37.309887 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:37.308999    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:16:37.309887 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.309467    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:37.309887 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:37.309577    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:16:37.344198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.344153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c72np" event={"ID":"a9af1130-d46a-415f-9086-c10df2dd7ad5","Type":"ContainerStarted","Data":"50df7d8e7a4e411392d4d5c645f38daaf76f79c04abd0642c0f94420faacd507"}
Sep 09 20:16:37.351068 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.351032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jx5cd" event={"ID":"73d9e6c4-c63a-458e-a181-def2c4405b45","Type":"ContainerStarted","Data":"bfaa0d45901ee4b2de0d542ef18bec01dfd2c96597e4078e354e96d113df5868"}
Sep 09 20:16:37.367693 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.367638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl" event={"ID":"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277","Type":"ContainerStarted","Data":"bf63b4ce830c0ca14b89664f4b3ffa4058069e754b9ff05623d4d5c219d5587c"}
Sep 09 20:16:37.385569 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.385529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-klsl6" event={"ID":"8696794c-360f-4af6-bc12-81b6c8f58ea5","Type":"ContainerStarted","Data":"9e071412cae7073cf9c903a819fd281cf91590225505bb8ade73906d2dc28eff"}
Sep 09 20:16:37.391125 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.391084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t64d7" event={"ID":"36251d92-c8c9-450d-ac2c-103c4d5d38e6","Type":"ContainerStarted","Data":"8a86ab01d0bcb4282019091e2ea7ffef8f1f4ae52b28a91a09ba27a8f1e7914c"}
Sep 09 20:16:37.402037 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.401997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vdbpg" event={"ID":"be868ed8-2f22-414e-91f2-5e511d69b5b4","Type":"ContainerStarted","Data":"ee44a9778353f8ec289d5b2c35a92ff1de93e4579e50e4d2d1001d35471a40af"}
Sep 09 20:16:37.414206 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.414148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" event={"ID":"20ada389393df0af1b24f49af160350a","Type":"ContainerStarted","Data":"f535d4852cc494e5e035fa041e89ba6a996aaea9ea243330bdd4d513a0a8d9d7"}
Sep 09 20:16:37.425713 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.425099    2563 generic.go:356] "Generic (PLEG): container finished" podID="9a6999f592201b6868f277532693fdb1" containerID="caf3e861bccbdcdababfbe0a58741fd9f116d005b534b804280ae3b2abaef9af" exitCode=0
Sep 09 20:16:37.425713 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.425202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" event={"ID":"9a6999f592201b6868f277532693fdb1","Type":"ContainerDied","Data":"caf3e861bccbdcdababfbe0a58741fd9f116d005b534b804280ae3b2abaef9af"}
Sep 09 20:16:37.432556 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.432399    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" podStartSLOduration=2.432374786 podStartE2EDuration="2.432374786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:37.431873182 +0000 UTC m=+4.670975487" watchObservedRunningTime="2026-09-09 20:16:37.432374786 +0000 UTC m=+4.671477101"
Sep 09 20:16:37.433578 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.433347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t" event={"ID":"faac4705-5d14-4b37-8a0f-a03925b24e5c","Type":"ContainerStarted","Data":"e2b0ffc50c5cb72ca34749e250f0b0e476d45057bb0545ae878866f474793b81"}
Sep 09 20:16:37.466588 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.462611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h4vr2" event={"ID":"29247776-5cf6-420f-897b-18d0d9ae3a7a","Type":"ContainerStarted","Data":"469255b218b5076391c84aaaea22cd9cf827169a014bf3036491ddac56d518d4"}
Sep 09 20:16:37.485834 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:37.485726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zsfbn" event={"ID":"1833449f-dfe7-4c13-ae92-562b409d1cf6","Type":"ContainerStarted","Data":"d44db6a7a94e75ee89623c430aaa415209bb5faab57e2360809c27fe5be39345"}
Sep 09 20:16:38.307628 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:38.307044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:38.307628 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:38.307186    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:16:38.511763 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:38.511720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" event={"ID":"9a6999f592201b6868f277532693fdb1","Type":"ContainerStarted","Data":"864c2200ebcee81f2a0f7a2b61e42daebd1d71e882365c6bfba7ef3744233c4d"}
Sep 09 20:16:38.796998 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:38.796229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs\") pod \"network-metrics-daemon-86bgz\" (UID: \"7377be15-1184-4239-a696-75be3f1d835d\") " pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:38.796998 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:38.796425    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:38.796998 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:38.796495    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs podName:7377be15-1184-4239-a696-75be3f1d835d nodeName:}" failed. No retries permitted until 2026-09-09 20:16:42.796476075 +0000 UTC m=+10.035578373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs") pod "network-metrics-daemon-86bgz" (UID: "7377be15-1184-4239-a696-75be3f1d835d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:38.898171 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:38.897406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:38.898171 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:38.897663    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:38.898171 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:38.897734    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret podName:d65fd73a-56b3-4e21-988d-cd2d26bb8847 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:42.897715055 +0000 UTC m=+10.136817336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret") pod "global-pull-secret-syncer-lsr8k" (UID: "d65fd73a-56b3-4e21-988d-cd2d26bb8847") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:38.998617 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:38.997890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfsrl\" (UniqueName: \"kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl\") pod \"network-check-target-pkqbv\" (UID: \"011ed84f-e0dd-495c-9eb3-fd0ad001c3d5\") " pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:38.998617 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:38.998097    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:16:38.998617 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:38.998119    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:16:38.998617 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:38.998133    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfsrl for pod openshift-network-diagnostics/network-check-target-pkqbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:38.998617 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:38.998205    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl podName:011ed84f-e0dd-495c-9eb3-fd0ad001c3d5 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:42.998186802 +0000 UTC m=+10.237289088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfsrl" (UniqueName: "kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl") pod "network-check-target-pkqbv" (UID: "011ed84f-e0dd-495c-9eb3-fd0ad001c3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:39.312961 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:39.312921    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:39.313499 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:39.313142    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:16:39.313693 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:39.313672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:39.313835 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:39.313814    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:16:40.307457 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:40.307083    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:40.307457 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:40.307243    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:16:41.310389 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:41.310045    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:41.310389 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:41.310080    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:41.310389 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:41.310197    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:16:41.310389 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:41.310343    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:16:42.307978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:42.307941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:42.308189 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:42.308096    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:16:42.833926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:42.833391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs\") pod \"network-metrics-daemon-86bgz\" (UID: \"7377be15-1184-4239-a696-75be3f1d835d\") " pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:42.833926 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:42.833545    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:42.833926 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:42.833630    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs podName:7377be15-1184-4239-a696-75be3f1d835d nodeName:}" failed. No retries permitted until 2026-09-09 20:16:50.833594459 +0000 UTC m=+18.072696740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs") pod "network-metrics-daemon-86bgz" (UID: "7377be15-1184-4239-a696-75be3f1d835d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:42.934332 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:42.934289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:42.934512 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:42.934457    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:42.934593 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:42.934526    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret podName:d65fd73a-56b3-4e21-988d-cd2d26bb8847 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:50.934506771 +0000 UTC m=+18.173609065 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret") pod "global-pull-secret-syncer-lsr8k" (UID: "d65fd73a-56b3-4e21-988d-cd2d26bb8847") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:43.035667 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:43.035626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfsrl\" (UniqueName: \"kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl\") pod \"network-check-target-pkqbv\" (UID: \"011ed84f-e0dd-495c-9eb3-fd0ad001c3d5\") " pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:43.035912 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:43.035846    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:16:43.035912 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:43.035874    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:16:43.035912 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:43.035891    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfsrl for pod openshift-network-diagnostics/network-check-target-pkqbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:43.036133 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:43.035953    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl podName:011ed84f-e0dd-495c-9eb3-fd0ad001c3d5 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:51.035934132 +0000 UTC m=+18.275036418 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfsrl" (UniqueName: "kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl") pod "network-check-target-pkqbv" (UID: "011ed84f-e0dd-495c-9eb3-fd0ad001c3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:43.308890 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:43.308785    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:43.309054 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:43.308919    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:16:43.309387 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:43.309201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:43.309387 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:43.309331    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:16:44.307201 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:44.307100    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:44.307694 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:44.307222    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:16:45.308179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:45.307558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:45.308179 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:45.307774    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:16:45.308179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:45.307792    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:45.308179 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:45.307932    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:16:46.307788 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:46.307754    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:46.307941 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:46.307867    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:16:47.307964 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:47.307925    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:47.308404 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:47.307978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:47.308404 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:47.308081    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:16:47.308404 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:47.308162    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:16:48.307470 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:48.307433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:48.307663 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:48.307550    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:16:49.307386 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:49.307356    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:49.307852 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:49.307494    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:16:49.307852 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:49.307554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:49.307852 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:49.307681    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:16:50.308203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:50.307990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:50.308747 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:50.308307    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:16:50.892443 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:50.892402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs\") pod \"network-metrics-daemon-86bgz\" (UID: \"7377be15-1184-4239-a696-75be3f1d835d\") " pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:50.892670 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:50.892554    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:50.892670 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:50.892650    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs podName:7377be15-1184-4239-a696-75be3f1d835d nodeName:}" failed. No retries permitted until 2026-09-09 20:17:06.892626112 +0000 UTC m=+34.131728393 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs") pod "network-metrics-daemon-86bgz" (UID: "7377be15-1184-4239-a696-75be3f1d835d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:50.993773 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:50.993733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:50.993949 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:50.993899    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:50.994023 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:50.993968    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret podName:d65fd73a-56b3-4e21-988d-cd2d26bb8847 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:06.993951274 +0000 UTC m=+34.233053567 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret") pod "global-pull-secret-syncer-lsr8k" (UID: "d65fd73a-56b3-4e21-988d-cd2d26bb8847") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:51.094274 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:51.094225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfsrl\" (UniqueName: \"kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl\") pod \"network-check-target-pkqbv\" (UID: \"011ed84f-e0dd-495c-9eb3-fd0ad001c3d5\") " pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:51.094420 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:51.094369    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:16:51.094420 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:51.094387    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:16:51.094420 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:51.094399    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfsrl for pod openshift-network-diagnostics/network-check-target-pkqbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:51.094604 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:51.094464    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl podName:011ed84f-e0dd-495c-9eb3-fd0ad001c3d5 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:07.094445768 +0000 UTC m=+34.333548069 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfsrl" (UniqueName: "kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl") pod "network-check-target-pkqbv" (UID: "011ed84f-e0dd-495c-9eb3-fd0ad001c3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:51.307181 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:51.307139    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:51.307338 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:51.307139    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:51.307338 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:51.307275    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:16:51.307338 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:51.307317    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:16:52.307888 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:52.307852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:52.308311 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:52.307973    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:16:53.310900 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:53.310870    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:53.311315 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:53.310969    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:16:53.311315 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:53.311026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:53.311315 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:53.311142    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:16:54.307332 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:54.307295    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:54.307512 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:54.307405    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:16:55.311689 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.311474    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:55.312139 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.311524    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:55.312139 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:55.311772    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:16:55.312139 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:55.311851    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:16:55.548004 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.547967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t" event={"ID":"faac4705-5d14-4b37-8a0f-a03925b24e5c","Type":"ContainerStarted","Data":"929b83561b3dafa7ad6d279878a08eef0a7ab873fbf77f8e027de25feed9688a"}
Sep 09 20:16:55.549643 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.549613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zsfbn" event={"ID":"1833449f-dfe7-4c13-ae92-562b409d1cf6","Type":"ContainerStarted","Data":"13cbba04ed4b04c45e984f54b5967873ca8a403d38d1d921e17b274fcf39f63e"}
Sep 09 20:16:55.551189 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.551167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c72np" event={"ID":"a9af1130-d46a-415f-9086-c10df2dd7ad5","Type":"ContainerStarted","Data":"b92f8d6629cec203b44164f0a3f5285d6059d62bae7406e0bd1efb6b711a7938"}
Sep 09 20:16:55.552624 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.552602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jx5cd" event={"ID":"73d9e6c4-c63a-458e-a181-def2c4405b45","Type":"ContainerStarted","Data":"3873e07a6aa5aebed716b05f532cacec8ed7d7d87af23b6636a6c43eb97c0e9e"}
Sep 09 20:16:55.555610 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.555562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl" event={"ID":"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277","Type":"ContainerStarted","Data":"0ca7475c81a67559b841b9bc3be6a24c73474fd42c9b740e1e37f0362804e37f"}
Sep 09 20:16:55.555708 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.555642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl" event={"ID":"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277","Type":"ContainerStarted","Data":"111371aee03283252bc8fa4a8a867f729a8e943d92a251c978c5753eea5ce6f7"}
Sep 09 20:16:55.555708 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.555673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl" event={"ID":"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277","Type":"ContainerStarted","Data":"b000b0c48b98c077bc4a24effa7f077e19e45fc8ef5aa852c77bd1f1d3f64950"}
Sep 09 20:16:55.555708 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.555701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl" event={"ID":"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277","Type":"ContainerStarted","Data":"04ee6f6e3ef565e9a6987535347b30a9553544daaed8a4fa009de2ed93c6d302"}
Sep 09 20:16:55.555873 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.555727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl" event={"ID":"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277","Type":"ContainerStarted","Data":"413ae8f3c6d6563f7d8cfd2f05f31335b09ce30eb35abc2cdd9caac58bab1884"}
Sep 09 20:16:55.555873 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.555763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl" event={"ID":"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277","Type":"ContainerStarted","Data":"4cd497f6c0bef7ca828051653ff0ee1bdf4cc190be9ce1989db885f29319513a"}
Sep 09 20:16:55.556982 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.556961    2563 generic.go:356] "Generic (PLEG): container finished" podID="8696794c-360f-4af6-bc12-81b6c8f58ea5" containerID="9f1fbeaddc2a7a9e7fe8f9af725992c9bb2ed3bc3409834570959bd53fd82d4a" exitCode=0
Sep 09 20:16:55.557056 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.557020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-klsl6" event={"ID":"8696794c-360f-4af6-bc12-81b6c8f58ea5","Type":"ContainerDied","Data":"9f1fbeaddc2a7a9e7fe8f9af725992c9bb2ed3bc3409834570959bd53fd82d4a"}
Sep 09 20:16:55.558520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.558501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t64d7" event={"ID":"36251d92-c8c9-450d-ac2c-103c4d5d38e6","Type":"ContainerStarted","Data":"b879d6145da38cadee08233387826ffc9231b2491db8ee98587efbdfd5575223"}
Sep 09 20:16:55.560298 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.560276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vdbpg" event={"ID":"be868ed8-2f22-414e-91f2-5e511d69b5b4","Type":"ContainerStarted","Data":"fb5596e8337ee23e51120cfb734bd1845d654104c9437a6d2daaa9b7437c47ec"}
Sep 09 20:16:55.566242 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.566206    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zsfbn" podStartSLOduration=3.14785496 podStartE2EDuration="21.566194448s" totalImagesPullingTime="18.418339488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:55.565967035 +0000 UTC m=+22.805069335" watchObservedRunningTime="2026-09-09 20:16:55.566194448 +0000 UTC m=+22.805296750"
Sep 09 20:16:55.566330 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.566281    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" podStartSLOduration=20.566276651 podStartE2EDuration="20.566276651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:38.529821769 +0000 UTC m=+5.768924074" watchObservedRunningTime="2026-09-09 20:16:55.566276651 +0000 UTC m=+22.805378951"
Sep 09 20:16:55.581162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.581120    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t64d7" podStartSLOduration=3.200587822 podStartE2EDuration="21.581105991s" totalImagesPullingTime="18.380518169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:55.580630323 +0000 UTC m=+22.819732622" watchObservedRunningTime="2026-09-09 20:16:55.581105991 +0000 UTC m=+22.820208290"
Sep 09 20:16:55.624650 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.624597    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c72np" podStartSLOduration=3.1360339489999998 podStartE2EDuration="21.624567529s" totalImagesPullingTime="18.48853358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:55.624066719 +0000 UTC m=+22.863169019" watchObservedRunningTime="2026-09-09 20:16:55.624567529 +0000 UTC m=+22.863669829"
Sep 09 20:16:55.641813 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.641769    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vdbpg" podStartSLOduration=3.203349073 podStartE2EDuration="21.641754742s" totalImagesPullingTime="18.438405669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:55.641441326 +0000 UTC m=+22.880543617" watchObservedRunningTime="2026-09-09 20:16:55.641754742 +0000 UTC m=+22.880857072"
Sep 09 20:16:55.659597 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.659540    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jx5cd" podStartSLOduration=3.286153625 podStartE2EDuration="21.659527135s" totalImagesPullingTime="18.37337351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:55.65931007 +0000 UTC m=+22.898412371" watchObservedRunningTime="2026-09-09 20:16:55.659527135 +0000 UTC m=+22.898629429"
Sep 09 20:16:55.925254 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:55.925128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:56.032582 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:56.032534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:56.307408 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:56.307375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:56.307589 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:56.307486    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:16:56.455471 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:56.455446    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:16:56.534167 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:56.534064    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jx5cd"
Sep 09 20:16:56.534789 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:56.534764    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jx5cd"
Sep 09 20:16:56.565073 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:56.565032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t" event={"ID":"faac4705-5d14-4b37-8a0f-a03925b24e5c","Type":"ContainerStarted","Data":"0afa4fe3cf6649a52d64773c1f28121a87829b6e01601be0c65357c2d6514122"}
Sep 09 20:16:56.567066 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:56.566964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h4vr2" event={"ID":"29247776-5cf6-420f-897b-18d0d9ae3a7a","Type":"ContainerStarted","Data":"2b755cf957334abaa18d83c4a90aaa018672f90ea078775efbce8b28d0806641"}
Sep 09 20:16:56.788526 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:56.788247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:56.860094 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:56.860042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:56.930411 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:56.930382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:57.216961 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:57.216783    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:16:56.455464315Z","UUID":"ca7bd8ad-a489-4859-8f6b-0028ca2a3d73","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:16:57.218557 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:57.218528    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:16:57.218698 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:57.218563    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:16:57.311457 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:57.311430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:57.311457 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:57.311446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:57.311714 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:57.311544    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:16:57.311714 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:57.311688    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:16:57.569233 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:57.569164    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 20:16:58.307429 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:58.307395    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:16:58.307638 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:58.307498    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:16:58.576903 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:58.576814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t" event={"ID":"faac4705-5d14-4b37-8a0f-a03925b24e5c","Type":"ContainerStarted","Data":"536c836c87f37bf187cdbd7be6484ed972e462d755b7067cfde5198e56174284"}
Sep 09 20:16:58.579930 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:58.579903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl" event={"ID":"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277","Type":"ContainerStarted","Data":"ae5718e8901717e9678859a9af86a48818e9a6f84b8c8b724acbc3ea9c65cedd"}
Sep 09 20:16:58.596252 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:58.596193    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h4vr2" podStartSLOduration=6.181609672 podStartE2EDuration="24.596173472s" totalImagesPullingTime="18.4145638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:56.602231311 +0000 UTC m=+23.841333611" watchObservedRunningTime="2026-09-09 20:16:58.596173472 +0000 UTC m=+25.835275774"
Sep 09 20:16:58.596612 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:58.596566    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4l6t" podStartSLOduration=3.796557846 podStartE2EDuration="24.596557086s" totalImagesPullingTime="20.79999924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:58.595746095 +0000 UTC m=+25.834848405" watchObservedRunningTime="2026-09-09 20:16:58.596557086 +0000 UTC m=+25.835659389"
Sep 09 20:16:58.649546 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:58.649500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:58.750319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:58.750263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:59.308035 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:59.307999    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:16:59.308035 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:59.308024    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:16:59.308245 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:59.308159    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:16:59.308287 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:16:59.308260    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:16:59.673236 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:59.673152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:59.774589 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:16:59.774544    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:00.296779 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:00.296737    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jx5cd"
Sep 09 20:17:00.296962 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:00.296880    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 20:17:00.297384 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:00.297361    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jx5cd"
Sep 09 20:17:00.307716 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:00.307685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:17:00.307900 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:00.307811    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:17:00.357262 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:00.357229    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:00.679126 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:00.679036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:01.158508 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:01.158338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:01.227146 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:01.227116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:01.310377 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:01.310343    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:17:01.310377 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:01.310362    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:17:01.310558 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:01.310456    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:17:01.310558 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:01.310499    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:17:01.590793 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:01.590704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl" event={"ID":"d73a1970-96aa-4bd3-b5cf-0c9e5a5c0277","Type":"ContainerStarted","Data":"d91a69be90711d3d4b651226948038d244bfc647540df5ff6bd1bb1ba0c08f1a"}
Sep 09 20:17:01.591097 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:01.591067    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:17:01.592561 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:01.592534    2563 generic.go:356] "Generic (PLEG): container finished" podID="8696794c-360f-4af6-bc12-81b6c8f58ea5" containerID="51985b4f0fbadeab44ea4596880bcd90bd72b5fcd8ed2d96c977bce2064f4192" exitCode=0
Sep 09 20:17:01.592683 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:01.592613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-klsl6" event={"ID":"8696794c-360f-4af6-bc12-81b6c8f58ea5","Type":"ContainerDied","Data":"51985b4f0fbadeab44ea4596880bcd90bd72b5fcd8ed2d96c977bce2064f4192"}
Sep 09 20:17:01.608993 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:01.608965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:17:01.667946 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:01.667892    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl" podStartSLOduration=9.203680744 podStartE2EDuration="27.667874249s" totalImagesPullingTime="18.464193505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:01.635183835 +0000 UTC m=+28.874286135" watchObservedRunningTime="2026-09-09 20:17:01.667874249 +0000 UTC m=+28.906976548"
Sep 09 20:17:02.307561 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:02.307529    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:17:02.308016 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:02.307638    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:17:02.595966 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:02.595931    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:17:02.596083 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:02.596025    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:17:02.611923 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:02.611898    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:17:02.751005 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:02.750971    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-86bgz"]
Sep 09 20:17:02.751180 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:02.751138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:17:02.751272 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:02.751251    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:17:02.753760 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:02.753734    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pkqbv"]
Sep 09 20:17:02.753877 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:02.753838    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:17:02.753956 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:02.753934    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:17:02.754422 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:02.754397    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lsr8k"]
Sep 09 20:17:02.754522 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:02.754478    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:17:02.754567 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:02.754547    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:17:03.600258 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:03.600056    2563 generic.go:356] "Generic (PLEG): container finished" podID="8696794c-360f-4af6-bc12-81b6c8f58ea5" containerID="d6072c1b4b794cfc02352bca59b1042f8843de92d506ac6d2343260db755bc6c" exitCode=0
Sep 09 20:17:03.600258 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:03.600139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-klsl6" event={"ID":"8696794c-360f-4af6-bc12-81b6c8f58ea5","Type":"ContainerDied","Data":"d6072c1b4b794cfc02352bca59b1042f8843de92d506ac6d2343260db755bc6c"}
Sep 09 20:17:04.307532 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:04.307502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:17:04.307669 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:04.307633    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:17:04.307807 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:04.307775    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:17:04.307908 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:04.307772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:17:04.307908 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:04.307871    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:17:04.308041 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:04.308014    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:17:04.605435 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:04.605349    2563 generic.go:356] "Generic (PLEG): container finished" podID="8696794c-360f-4af6-bc12-81b6c8f58ea5" containerID="bf9e69bcd13e7b9f6e8b9be7f98d638c8c447302c6cad534a9e8c4f935e3568e" exitCode=0
Sep 09 20:17:04.605878 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:04.605432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-klsl6" event={"ID":"8696794c-360f-4af6-bc12-81b6c8f58ea5","Type":"ContainerDied","Data":"bf9e69bcd13e7b9f6e8b9be7f98d638c8c447302c6cad534a9e8c4f935e3568e"}
Sep 09 20:17:05.330774 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:05.330738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:05.431021 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:05.430968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:06.013753 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:06.013705    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:06.097224 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:06.097176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:06.307378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:06.307303    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:17:06.307378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:06.307349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:17:06.307378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:06.307353    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:17:06.307606 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:06.307430    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bgz" podUID="7377be15-1184-4239-a696-75be3f1d835d"
Sep 09 20:17:06.307606 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:06.307500    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkqbv" podUID="011ed84f-e0dd-495c-9eb3-fd0ad001c3d5"
Sep 09 20:17:06.307606 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:06.307555    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lsr8k" podUID="d65fd73a-56b3-4e21-988d-cd2d26bb8847"
Sep 09 20:17:06.320294 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:06.320262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:06.416240 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:06.416205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:06.856547 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:06.856509    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:06.913524 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:06.913489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs\") pod \"network-metrics-daemon-86bgz\" (UID: \"7377be15-1184-4239-a696-75be3f1d835d\") " pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:17:06.913709 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:06.913623    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:17:06.913709 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:06.913676    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs podName:7377be15-1184-4239-a696-75be3f1d835d nodeName:}" failed. No retries permitted until 2026-09-09 20:17:38.91366126 +0000 UTC m=+66.152763538 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs") pod "network-metrics-daemon-86bgz" (UID: "7377be15-1184-4239-a696-75be3f1d835d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:17:06.956390 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:06.956352    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:07.014160 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:07.014127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:17:07.014627 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:07.014270    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:17:07.014627 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:07.014332    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret podName:d65fd73a-56b3-4e21-988d-cd2d26bb8847 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:39.014317288 +0000 UTC m=+66.253419566 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret") pod "global-pull-secret-syncer-lsr8k" (UID: "d65fd73a-56b3-4e21-988d-cd2d26bb8847") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:17:07.114979 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:07.114891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfsrl\" (UniqueName: \"kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl\") pod \"network-check-target-pkqbv\" (UID: \"011ed84f-e0dd-495c-9eb3-fd0ad001c3d5\") " pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:17:07.115129 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:07.115037    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:17:07.115129 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:07.115055    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:17:07.115129 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:07.115066    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfsrl for pod openshift-network-diagnostics/network-check-target-pkqbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:17:07.115259 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:07.115137    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl podName:011ed84f-e0dd-495c-9eb3-fd0ad001c3d5 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:39.115119235 +0000 UTC m=+66.354221537 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfsrl" (UniqueName: "kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl") pod "network-check-target-pkqbv" (UID: "011ed84f-e0dd-495c-9eb3-fd0ad001c3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:17:07.354632 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:07.354592    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zsfbn_1833449f-dfe7-4c13-ae92-562b409d1cf6/dns-node-resolver/0.log"
Sep 09 20:17:07.390286 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:07.390167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:07.458859 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:07.458814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:07.566462 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:07.566429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:07.667630 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:07.667531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:08.009849 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.009819    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeReady"
Sep 09 20:17:08.010014 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.009979    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:17:08.051138 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.051103    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bb4794f65-7rk9b"]
Sep 09 20:17:08.075568 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.075537    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ng7mg"]
Sep 09 20:17:08.075749 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.075686    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.078755 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.078721    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:17:08.079450 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.079424    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:17:08.079584 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.079426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:17:08.079953 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.079931    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qrtsj\""
Sep 09 20:17:08.090613 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.090399    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bb4794f65-7rk9b"]
Sep 09 20:17:08.090613 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.090445    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xvxgm"]
Sep 09 20:17:08.090806 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.090726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:08.090995 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.090963    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:17:08.093539 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.093516    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:17:08.093648 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.093541    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8pv6p\""
Sep 09 20:17:08.093695 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.093665    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:17:08.106314 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.106289    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ng7mg"]
Sep 09 20:17:08.106413 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.106329    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xvxgm"]
Sep 09 20:17:08.106413 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.106362    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dcf7r"]
Sep 09 20:17:08.106597 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.106498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xvxgm"
Sep 09 20:17:08.109515 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.109499    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:17:08.110790 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.109515    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:17:08.110790 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.109557    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:17:08.110790 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.109702    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mt874\""
Sep 09 20:17:08.125086 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.125062    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dcf7r"]
Sep 09 20:17:08.125210 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.125195    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.128865 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.128839    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:17:08.128999 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.128881    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s82pj\""
Sep 09 20:17:08.128999 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.128944    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:17:08.129181 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.129135    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:17:08.129181 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.129148    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:17:08.151638 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.151613    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t64d7_36251d92-c8c9-450d-ac2c-103c4d5d38e6/node-ca/0.log"
Sep 09 20:17:08.200317 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.200220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:08.221703 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.221666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.221865 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.221708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/519bb8de-4eec-41ca-b66c-122e0bf314ea-installation-pull-secrets\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.221865 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.221735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-bound-sa-token\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.221865 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.221756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgqc\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-kube-api-access-9vgqc\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.221865 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.221803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-certificates\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.221865 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.221828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/519bb8de-4eec-41ca-b66c-122e0bf314ea-image-registry-private-configuration\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.221865 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.221854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d384213-5569-4654-bb4f-6e1834675c73-data-volume\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.222111 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.221922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fn7\" (UniqueName: \"kubernetes.io/projected/4d384213-5569-4654-bb4f-6e1834675c73-kube-api-access-m9fn7\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.222111 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.221959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.222111 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.222029    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-config-volume\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:08.222111 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.222053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-tmp-dir\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:08.222111 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.222082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/519bb8de-4eec-41ca-b66c-122e0bf314ea-ca-trust-extracted\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.222111 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.222106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/519bb8de-4eec-41ca-b66c-122e0bf314ea-trusted-ca\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.222341 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.222140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3946c37-c56f-44b5-82ab-5a1b617fb2b1-cert\") pod \"ingress-canary-xvxgm\" (UID: \"d3946c37-c56f-44b5-82ab-5a1b617fb2b1\") " pod="openshift-ingress-canary/ingress-canary-xvxgm"
Sep 09 20:17:08.222341 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.222183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4d384213-5569-4654-bb4f-6e1834675c73-crio-socket\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.222341 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.222285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrltd\" (UniqueName: \"kubernetes.io/projected/d3946c37-c56f-44b5-82ab-5a1b617fb2b1-kube-api-access-mrltd\") pod \"ingress-canary-xvxgm\" (UID: \"d3946c37-c56f-44b5-82ab-5a1b617fb2b1\") " pod="openshift-ingress-canary/ingress-canary-xvxgm"
Sep 09 20:17:08.222341 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.222326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.222490 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.222371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65msp\" (UniqueName: \"kubernetes.io/projected/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-kube-api-access-65msp\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:08.222490 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.222393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:08.307091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.307054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:17:08.307281 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.307175    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:17:08.307281 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.307192    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:17:08.311346 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.311294    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:17:08.311507 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.311373    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:17:08.311507 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.311422    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c88hn\""
Sep 09 20:17:08.311507 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.311432    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wv6m4\""
Sep 09 20:17:08.311507 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.311424    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:17:08.311758 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.311742    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:17:08.322785 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.322759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d384213-5569-4654-bb4f-6e1834675c73-data-volume\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.322908 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.322798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fn7\" (UniqueName: \"kubernetes.io/projected/4d384213-5569-4654-bb4f-6e1834675c73-kube-api-access-m9fn7\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.322908 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.322822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.323010 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.322931    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:17:08.323010 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.322945    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb4794f65-7rk9b: secret "image-registry-tls" not found
Sep 09 20:17:08.323010 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.322952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-config-volume\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:08.323010 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.322988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-tmp-dir\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:08.323010 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.323008    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls podName:519bb8de-4eec-41ca-b66c-122e0bf314ea nodeName:}" failed. No retries permitted until 2026-09-09 20:17:08.822988806 +0000 UTC m=+36.062091107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls") pod "image-registry-7bb4794f65-7rk9b" (UID: "519bb8de-4eec-41ca-b66c-122e0bf314ea") : secret "image-registry-tls" not found
Sep 09 20:17:08.323261 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/519bb8de-4eec-41ca-b66c-122e0bf314ea-ca-trust-extracted\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.323261 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/519bb8de-4eec-41ca-b66c-122e0bf314ea-trusted-ca\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.323261 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3946c37-c56f-44b5-82ab-5a1b617fb2b1-cert\") pod \"ingress-canary-xvxgm\" (UID: \"d3946c37-c56f-44b5-82ab-5a1b617fb2b1\") " pod="openshift-ingress-canary/ingress-canary-xvxgm"
Sep 09 20:17:08.323261 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d384213-5569-4654-bb4f-6e1834675c73-data-volume\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.323261 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4d384213-5569-4654-bb4f-6e1834675c73-crio-socket\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.323261 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrltd\" (UniqueName: \"kubernetes.io/projected/d3946c37-c56f-44b5-82ab-5a1b617fb2b1-kube-api-access-mrltd\") pod \"ingress-canary-xvxgm\" (UID: \"d3946c37-c56f-44b5-82ab-5a1b617fb2b1\") " pod="openshift-ingress-canary/ingress-canary-xvxgm"
Sep 09 20:17:08.323261 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.323261 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.323220    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:17:08.323261 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.323256    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm podName:4d384213-5569-4654-bb4f-6e1834675c73 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:08.823248663 +0000 UTC m=+36.062350941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dcf7r" (UID: "4d384213-5569-4654-bb4f-6e1834675c73") : configmap "kube-rbac-proxy" not found
Sep 09 20:17:08.323722 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4d384213-5569-4654-bb4f-6e1834675c73-crio-socket\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.323722 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65msp\" (UniqueName: \"kubernetes.io/projected/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-kube-api-access-65msp\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:08.323722 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:08.323722 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.323722 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/519bb8de-4eec-41ca-b66c-122e0bf314ea-installation-pull-secrets\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.323722 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-bound-sa-token\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.323722 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/519bb8de-4eec-41ca-b66c-122e0bf314ea-ca-trust-extracted\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.323722 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgqc\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-kube-api-access-9vgqc\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.323722 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-certificates\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.323722 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/519bb8de-4eec-41ca-b66c-122e0bf314ea-image-registry-private-configuration\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.323722 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.323676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-config-volume\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:08.324304 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.324060    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:17:08.324304 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.324122    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls podName:e51731de-fe8e-4ea7-a1f9-7661ebab8c1a nodeName:}" failed. No retries permitted until 2026-09-09 20:17:08.824104007 +0000 UTC m=+36.063206300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls") pod "dns-default-ng7mg" (UID: "e51731de-fe8e-4ea7-a1f9-7661ebab8c1a") : secret "dns-default-metrics-tls" not found
Sep 09 20:17:08.324304 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.324163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-certificates\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.324304 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.324198    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:08.324304 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.324212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/519bb8de-4eec-41ca-b66c-122e0bf314ea-trusted-ca\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.324304 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.324235    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls podName:4d384213-5569-4654-bb4f-6e1834675c73 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:08.824224078 +0000 UTC m=+36.063326362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dcf7r" (UID: "4d384213-5569-4654-bb4f-6e1834675c73") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:08.324304 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.324303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-tmp-dir\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:08.328423 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.328398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/519bb8de-4eec-41ca-b66c-122e0bf314ea-installation-pull-secrets\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.328523 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.328432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/519bb8de-4eec-41ca-b66c-122e0bf314ea-image-registry-private-configuration\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.328592 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.328529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3946c37-c56f-44b5-82ab-5a1b617fb2b1-cert\") pod \"ingress-canary-xvxgm\" (UID: \"d3946c37-c56f-44b5-82ab-5a1b617fb2b1\") " pod="openshift-ingress-canary/ingress-canary-xvxgm"
Sep 09 20:17:08.365376 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.365345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrltd\" (UniqueName: \"kubernetes.io/projected/d3946c37-c56f-44b5-82ab-5a1b617fb2b1-kube-api-access-mrltd\") pod \"ingress-canary-xvxgm\" (UID: \"d3946c37-c56f-44b5-82ab-5a1b617fb2b1\") " pod="openshift-ingress-canary/ingress-canary-xvxgm"
Sep 09 20:17:08.365543 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.365438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgqc\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-kube-api-access-9vgqc\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.366474 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.366445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fn7\" (UniqueName: \"kubernetes.io/projected/4d384213-5569-4654-bb4f-6e1834675c73-kube-api-access-m9fn7\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.369299 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.369272    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-bound-sa-token\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.369858 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.369753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65msp\" (UniqueName: \"kubernetes.io/projected/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-kube-api-access-65msp\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:08.417382 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.417348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xvxgm"
Sep 09 20:17:08.527122 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.527072    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:08.614548 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.614403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:08.827428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.827336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:08.827428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.827383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.827698 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.827436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:08.827698 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.827501    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:17:08.827698 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.827550    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:08.827698 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.827561    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:17:08.827698 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.827595    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls podName:e51731de-fe8e-4ea7-a1f9-7661ebab8c1a nodeName:}" failed. No retries permitted until 2026-09-09 20:17:09.827555753 +0000 UTC m=+37.066658048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls") pod "dns-default-ng7mg" (UID: "e51731de-fe8e-4ea7-a1f9-7661ebab8c1a") : secret "dns-default-metrics-tls" not found
Sep 09 20:17:08.827698 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:08.827503    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:08.827698 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.827624    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls podName:4d384213-5569-4654-bb4f-6e1834675c73 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:09.827611381 +0000 UTC m=+37.066713659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dcf7r" (UID: "4d384213-5569-4654-bb4f-6e1834675c73") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:08.827698 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.827652    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm podName:4d384213-5569-4654-bb4f-6e1834675c73 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:09.827643315 +0000 UTC m=+37.066745602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dcf7r" (UID: "4d384213-5569-4654-bb4f-6e1834675c73") : configmap "kube-rbac-proxy" not found
Sep 09 20:17:08.827698 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.827662    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:17:08.827698 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.827680    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb4794f65-7rk9b: secret "image-registry-tls" not found
Sep 09 20:17:08.828083 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:08.827741    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls podName:519bb8de-4eec-41ca-b66c-122e0bf314ea nodeName:}" failed. No retries permitted until 2026-09-09 20:17:09.827723666 +0000 UTC m=+37.066825959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls") pod "image-registry-7bb4794f65-7rk9b" (UID: "519bb8de-4eec-41ca-b66c-122e0bf314ea") : secret "image-registry-tls" not found
Sep 09 20:17:09.153811 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.153719    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-p5p85"]
Sep 09 20:17:09.173063 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.173033    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-p5p85"]
Sep 09 20:17:09.173207 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.173162    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-p5p85"
Sep 09 20:17:09.175921 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.175897    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8jd4k\""
Sep 09 20:17:09.176081 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.175955    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 20:17:09.177078 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.176878    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 20:17:09.177078 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.176934    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 20:17:09.177243 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.177153    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 20:17:09.198101 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.198063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:09.260446 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.260400    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:09.332867 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.332834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/07f882d5-f61d-421c-b47e-7d57d11572ef-signing-key\") pod \"service-ca-75d99c484-p5p85\" (UID: \"07f882d5-f61d-421c-b47e-7d57d11572ef\") " pod="openshift-service-ca/service-ca-75d99c484-p5p85"
Sep 09 20:17:09.333042 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.332890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st47j\" (UniqueName: \"kubernetes.io/projected/07f882d5-f61d-421c-b47e-7d57d11572ef-kube-api-access-st47j\") pod \"service-ca-75d99c484-p5p85\" (UID: \"07f882d5-f61d-421c-b47e-7d57d11572ef\") " pod="openshift-service-ca/service-ca-75d99c484-p5p85"
Sep 09 20:17:09.333102 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.333039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/07f882d5-f61d-421c-b47e-7d57d11572ef-signing-cabundle\") pod \"service-ca-75d99c484-p5p85\" (UID: \"07f882d5-f61d-421c-b47e-7d57d11572ef\") " pod="openshift-service-ca/service-ca-75d99c484-p5p85"
Sep 09 20:17:09.434364 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.434285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st47j\" (UniqueName: \"kubernetes.io/projected/07f882d5-f61d-421c-b47e-7d57d11572ef-kube-api-access-st47j\") pod \"service-ca-75d99c484-p5p85\" (UID: \"07f882d5-f61d-421c-b47e-7d57d11572ef\") " pod="openshift-service-ca/service-ca-75d99c484-p5p85"
Sep 09 20:17:09.434518 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.434392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/07f882d5-f61d-421c-b47e-7d57d11572ef-signing-cabundle\") pod \"service-ca-75d99c484-p5p85\" (UID: \"07f882d5-f61d-421c-b47e-7d57d11572ef\") " pod="openshift-service-ca/service-ca-75d99c484-p5p85"
Sep 09 20:17:09.434518 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.434461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/07f882d5-f61d-421c-b47e-7d57d11572ef-signing-key\") pod \"service-ca-75d99c484-p5p85\" (UID: \"07f882d5-f61d-421c-b47e-7d57d11572ef\") " pod="openshift-service-ca/service-ca-75d99c484-p5p85"
Sep 09 20:17:09.435040 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.435014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/07f882d5-f61d-421c-b47e-7d57d11572ef-signing-cabundle\") pod \"service-ca-75d99c484-p5p85\" (UID: \"07f882d5-f61d-421c-b47e-7d57d11572ef\") " pod="openshift-service-ca/service-ca-75d99c484-p5p85"
Sep 09 20:17:09.437121 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.437098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/07f882d5-f61d-421c-b47e-7d57d11572ef-signing-key\") pod \"service-ca-75d99c484-p5p85\" (UID: \"07f882d5-f61d-421c-b47e-7d57d11572ef\") " pod="openshift-service-ca/service-ca-75d99c484-p5p85"
Sep 09 20:17:09.445058 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.445037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st47j\" (UniqueName: \"kubernetes.io/projected/07f882d5-f61d-421c-b47e-7d57d11572ef-kube-api-access-st47j\") pod \"service-ca-75d99c484-p5p85\" (UID: \"07f882d5-f61d-421c-b47e-7d57d11572ef\") " pod="openshift-service-ca/service-ca-75d99c484-p5p85"
Sep 09 20:17:09.484087 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.484048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-p5p85"
Sep 09 20:17:09.791175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.791142    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-p5p85"]
Sep 09 20:17:09.793879 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.793857    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xvxgm"]
Sep 09 20:17:09.795356 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:17:09.795332    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f882d5_f61d_421c_b47e_7d57d11572ef.slice/crio-76809b71f34157613a54062bd410570e09b0f70eb97df16135524e04cf490470 WatchSource:0}: Error finding container 76809b71f34157613a54062bd410570e09b0f70eb97df16135524e04cf490470: Status 404 returned error can't find the container with id 76809b71f34157613a54062bd410570e09b0f70eb97df16135524e04cf490470
Sep 09 20:17:09.796848 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:17:09.796826    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3946c37_c56f_44b5_82ab_5a1b617fb2b1.slice/crio-8f7ed33d14def90837cd0d6c308f1c9ca8a30fc650c8dca2cd2a16c8325da85f WatchSource:0}: Error finding container 8f7ed33d14def90837cd0d6c308f1c9ca8a30fc650c8dca2cd2a16c8325da85f: Status 404 returned error can't find the container with id 8f7ed33d14def90837cd0d6c308f1c9ca8a30fc650c8dca2cd2a16c8325da85f
Sep 09 20:17:09.838777 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.838749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:09.838875 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.838829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:09.838875 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.838855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:09.838875 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:09.838871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:09.838972 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:09.838903    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:17:09.838972 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:09.838924    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb4794f65-7rk9b: secret "image-registry-tls" not found
Sep 09 20:17:09.838972 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:09.838941    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:17:09.839067 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:09.838985    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:09.839067 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:09.839009    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:17:09.839067 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:09.838987    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls podName:519bb8de-4eec-41ca-b66c-122e0bf314ea nodeName:}" failed. No retries permitted until 2026-09-09 20:17:11.838968366 +0000 UTC m=+39.078070663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls") pod "image-registry-7bb4794f65-7rk9b" (UID: "519bb8de-4eec-41ca-b66c-122e0bf314ea") : secret "image-registry-tls" not found
Sep 09 20:17:09.839162 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:09.839086    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm podName:4d384213-5569-4654-bb4f-6e1834675c73 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:11.839068341 +0000 UTC m=+39.078170618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dcf7r" (UID: "4d384213-5569-4654-bb4f-6e1834675c73") : configmap "kube-rbac-proxy" not found
Sep 09 20:17:09.839162 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:09.839107    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls podName:4d384213-5569-4654-bb4f-6e1834675c73 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:11.839097735 +0000 UTC m=+39.078200013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dcf7r" (UID: "4d384213-5569-4654-bb4f-6e1834675c73") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:09.839162 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:09.839123    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls podName:e51731de-fe8e-4ea7-a1f9-7661ebab8c1a nodeName:}" failed. No retries permitted until 2026-09-09 20:17:11.839115128 +0000 UTC m=+39.078217406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls") pod "dns-default-ng7mg" (UID: "e51731de-fe8e-4ea7-a1f9-7661ebab8c1a") : secret "dns-default-metrics-tls" not found
Sep 09 20:17:10.101340 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:10.101303    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:10.621715 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:10.621671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-p5p85" event={"ID":"07f882d5-f61d-421c-b47e-7d57d11572ef","Type":"ContainerStarted","Data":"76809b71f34157613a54062bd410570e09b0f70eb97df16135524e04cf490470"}
Sep 09 20:17:10.624245 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:10.624156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xvxgm" event={"ID":"d3946c37-c56f-44b5-82ab-5a1b617fb2b1","Type":"ContainerStarted","Data":"8f7ed33d14def90837cd0d6c308f1c9ca8a30fc650c8dca2cd2a16c8325da85f"}
Sep 09 20:17:10.629633 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:10.629556    2563 generic.go:356] "Generic (PLEG): container finished" podID="8696794c-360f-4af6-bc12-81b6c8f58ea5" containerID="fd8b8fcdb366f2d5c354ade1f97c6a52d9a53066487daf360b878244db9012ad" exitCode=0
Sep 09 20:17:10.629633 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:10.629619    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-klsl6" event={"ID":"8696794c-360f-4af6-bc12-81b6c8f58ea5","Type":"ContainerDied","Data":"fd8b8fcdb366f2d5c354ade1f97c6a52d9a53066487daf360b878244db9012ad"}
Sep 09 20:17:10.794387 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:10.794345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:10.905249 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:10.905117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:11.563674 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:11.563634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:11.635872 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:11.635844    2563 generic.go:356] "Generic (PLEG): container finished" podID="8696794c-360f-4af6-bc12-81b6c8f58ea5" containerID="977a3b5209bf0e90fcd635f6a16f0f6b0266953c8a3d5da80a1b648da51ee4b8" exitCode=0
Sep 09 20:17:11.635872 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:11.635879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-klsl6" event={"ID":"8696794c-360f-4af6-bc12-81b6c8f58ea5","Type":"ContainerDied","Data":"977a3b5209bf0e90fcd635f6a16f0f6b0266953c8a3d5da80a1b648da51ee4b8"}
Sep 09 20:17:11.857629 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:11.857526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:11.857796 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:11.857633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:11.857796 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:11.857668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:11.857796 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:11.857689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:11.857796 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:11.857703    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:17:11.857796 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:11.857728    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb4794f65-7rk9b: secret "image-registry-tls" not found
Sep 09 20:17:11.857796 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:11.857757    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:17:11.857796 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:11.857786    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls podName:519bb8de-4eec-41ca-b66c-122e0bf314ea nodeName:}" failed. No retries permitted until 2026-09-09 20:17:15.857767828 +0000 UTC m=+43.096870120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls") pod "image-registry-7bb4794f65-7rk9b" (UID: "519bb8de-4eec-41ca-b66c-122e0bf314ea") : secret "image-registry-tls" not found
Sep 09 20:17:11.857796 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:11.857793    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:17:11.858102 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:11.857811    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm podName:4d384213-5569-4654-bb4f-6e1834675c73 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:15.857795885 +0000 UTC m=+43.096898163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dcf7r" (UID: "4d384213-5569-4654-bb4f-6e1834675c73") : configmap "kube-rbac-proxy" not found
Sep 09 20:17:11.858102 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:11.857833    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls podName:e51731de-fe8e-4ea7-a1f9-7661ebab8c1a nodeName:}" failed. No retries permitted until 2026-09-09 20:17:15.857823678 +0000 UTC m=+43.096925960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls") pod "dns-default-ng7mg" (UID: "e51731de-fe8e-4ea7-a1f9-7661ebab8c1a") : secret "dns-default-metrics-tls" not found
Sep 09 20:17:11.858102 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:11.857793    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:11.858102 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:11.857871    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls podName:4d384213-5569-4654-bb4f-6e1834675c73 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:15.857860566 +0000 UTC m=+43.096962844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dcf7r" (UID: "4d384213-5569-4654-bb4f-6e1834675c73") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:13.389255 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:13.389220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:13.486963 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:13.486913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:13.642886 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:13.642798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-p5p85" event={"ID":"07f882d5-f61d-421c-b47e-7d57d11572ef","Type":"ContainerStarted","Data":"37ded0a8cf864b78917f214945295a5f2f9f36a5807b2b94254762267380b8b3"}
Sep 09 20:17:13.644445 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:13.644419    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xvxgm" event={"ID":"d3946c37-c56f-44b5-82ab-5a1b617fb2b1","Type":"ContainerStarted","Data":"0de64a0084e02c32b3fd1ac14acdf7f90cf2aed4a44d5a275bafa769c5c27576"}
Sep 09 20:17:13.648134 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:13.648103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-klsl6" event={"ID":"8696794c-360f-4af6-bc12-81b6c8f58ea5","Type":"ContainerStarted","Data":"8ff198f7d241a531cb088622f63cb1a29a174d448d41343b414db08784573388"}
Sep 09 20:17:13.661815 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:13.661764    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-p5p85" podStartSLOduration=1.176916267 podStartE2EDuration="4.661748347s" totalImagesPullingTime="3.48483208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:13.661040052 +0000 UTC m=+40.900142353" watchObservedRunningTime="2026-09-09 20:17:13.661748347 +0000 UTC m=+40.900850643"
Sep 09 20:17:13.683435 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:13.683369    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-klsl6" podStartSLOduration=9.255783897 podStartE2EDuration="39.683351699s" totalImagesPullingTime="30.427567802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:13.681944944 +0000 UTC m=+40.921047243" watchObservedRunningTime="2026-09-09 20:17:13.683351699 +0000 UTC m=+40.922453996"
Sep 09 20:17:13.699097 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:13.699042    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xvxgm" podStartSLOduration=2.209363378 podStartE2EDuration="5.699026056s" totalImagesPullingTime="3.489662678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:13.698222731 +0000 UTC m=+40.937325031" watchObservedRunningTime="2026-09-09 20:17:13.699026056 +0000 UTC m=+40.938128355"
Sep 09 20:17:13.723906 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:13.723870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:14.613299 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:14.613261    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:14.706237 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:14.706199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:14.863695 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:14.863603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:14.967805 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:14.967773    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:15.038170 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:15.038126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:15.218595 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:15.218546    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:15.323281 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:15.323234    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:15.478028 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:15.477938    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:15.881289 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:15.881207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:15.894378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:15.894351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:15.894497 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:15.894401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:15.894497 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:15.894428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:15.894497 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:15.894486    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:17:15.894638 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:15.894511    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:17:15.894638 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:15.894518    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:15.894638 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:15.894485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:15.894638 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:15.894559    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm podName:4d384213-5569-4654-bb4f-6e1834675c73 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:23.894538588 +0000 UTC m=+51.133640882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dcf7r" (UID: "4d384213-5569-4654-bb4f-6e1834675c73") : configmap "kube-rbac-proxy" not found
Sep 09 20:17:15.894638 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:15.894563    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:17:15.894638 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:15.894594    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb4794f65-7rk9b: secret "image-registry-tls" not found
Sep 09 20:17:15.894638 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:15.894596    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls podName:e51731de-fe8e-4ea7-a1f9-7661ebab8c1a nodeName:}" failed. No retries permitted until 2026-09-09 20:17:23.894568805 +0000 UTC m=+51.133671083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls") pod "dns-default-ng7mg" (UID: "e51731de-fe8e-4ea7-a1f9-7661ebab8c1a") : secret "dns-default-metrics-tls" not found
Sep 09 20:17:15.894638 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:15.894613    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls podName:4d384213-5569-4654-bb4f-6e1834675c73 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:23.894605246 +0000 UTC m=+51.133707524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dcf7r" (UID: "4d384213-5569-4654-bb4f-6e1834675c73") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:15.894903 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:15.894671    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls podName:519bb8de-4eec-41ca-b66c-122e0bf314ea nodeName:}" failed. No retries permitted until 2026-09-09 20:17:23.894656123 +0000 UTC m=+51.133758401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls") pod "image-registry-7bb4794f65-7rk9b" (UID: "519bb8de-4eec-41ca-b66c-122e0bf314ea") : secret "image-registry-tls" not found
Sep 09 20:17:16.683842 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:16.683801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:16.889420 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:16.889385    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:17.092411 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:17.092369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:17.484105 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:17.484067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:17.894868 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:17.894776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:18.285879 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:18.285846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:18.498040 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:18.498003    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:18.735084 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:18.735038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:19.135680 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:19.135589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:19.210138 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:19.210102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:19.277377 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:19.277344    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:19.351235 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:19.351185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:19.482975 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:19.482940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:19.740472 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:19.740397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:20.682859 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:20.682809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:20.783294 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:20.783250    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:20.887902 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:20.887868    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:21.077758 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:21.077722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:21.227905 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:21.227871    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:21.494912 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:21.494876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:22.127215 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:22.127176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:22.484229 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:22.484186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:22.689173 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:22.689108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:23.280402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:23.280366    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:23.883966 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:23.883931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:23.957395 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:23.957357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:23.957598 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:23.957398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:23.957598 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:23.957451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:23.957598 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:23.957561    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:23.957767 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:23.957631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:23.957767 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:17:23.957651    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls podName:4d384213-5569-4654-bb4f-6e1834675c73 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:39.957630686 +0000 UTC m=+67.196732971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dcf7r" (UID: "4d384213-5569-4654-bb4f-6e1834675c73") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:23.958240 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:23.958213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d384213-5569-4654-bb4f-6e1834675c73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:23.960407 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:23.960383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls\") pod \"image-registry-7bb4794f65-7rk9b\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") " pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:23.960499 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:23.960380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e51731de-fe8e-4ea7-a1f9-7661ebab8c1a-metrics-tls\") pod \"dns-default-ng7mg\" (UID: \"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a\") " pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:23.988181 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:23.988141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:24.000284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:24.000253    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:24.107433 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:24.107397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:24.286346 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:24.286314    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bb4794f65-7rk9b"]
Sep 09 20:17:24.289252 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:17:24.289215    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519bb8de_4eec_41ca_b66c_122e0bf314ea.slice/crio-9895e52eeb4c7536880e34c94e8eb5c8bb11579b272b448116c79bb008633bbe WatchSource:0}: Error finding container 9895e52eeb4c7536880e34c94e8eb5c8bb11579b272b448116c79bb008633bbe: Status 404 returned error can't find the container with id 9895e52eeb4c7536880e34c94e8eb5c8bb11579b272b448116c79bb008633bbe
Sep 09 20:17:24.304765 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:24.304731    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:24.304979 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:24.304958    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ng7mg"]
Sep 09 20:17:24.307763 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:17:24.307741    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51731de_fe8e_4ea7_a1f9_7661ebab8c1a.slice/crio-c0960b5634bcdcb45ef3b944beb9612d32c67a881c12d7af55e7d7a5c8ebfdfd WatchSource:0}: Error finding container c0960b5634bcdcb45ef3b944beb9612d32c67a881c12d7af55e7d7a5c8ebfdfd: Status 404 returned error can't find the container with id c0960b5634bcdcb45ef3b944beb9612d32c67a881c12d7af55e7d7a5c8ebfdfd
Sep 09 20:17:24.476901 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:24.476858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:24.676811 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:24.676719    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ng7mg" event={"ID":"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a","Type":"ContainerStarted","Data":"c0960b5634bcdcb45ef3b944beb9612d32c67a881c12d7af55e7d7a5c8ebfdfd"}
Sep 09 20:17:24.678428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:24.678403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b" event={"ID":"519bb8de-4eec-41ca-b66c-122e0bf314ea","Type":"ContainerStarted","Data":"ca5811ecb2df0b4c570731e15c58980e36f2b59780c22ae1f813169d230ec7d1"}
Sep 09 20:17:24.678543 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:24.678441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b" event={"ID":"519bb8de-4eec-41ca-b66c-122e0bf314ea","Type":"ContainerStarted","Data":"9895e52eeb4c7536880e34c94e8eb5c8bb11579b272b448116c79bb008633bbe"}
Sep 09 20:17:24.678608 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:24.678545    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:24.698829 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:24.698781    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b" podStartSLOduration=37.698765434 podStartE2EDuration="37.698765434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:24.698115157 +0000 UTC m=+51.937217458" watchObservedRunningTime="2026-09-09 20:17:24.698765434 +0000 UTC m=+51.937867734"
Sep 09 20:17:25.081763 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:25.081725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:25.298095 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:25.298065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:25.779748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:25.779709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:25.879879 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:25.879828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:26.686140 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:26.686109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ng7mg" event={"ID":"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a","Type":"ContainerStarted","Data":"8bb817d441b1c047404786d8faaff538d715aef7bcc53fbef7ff98f7db8d3d2e"}
Sep 09 20:17:27.690396 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:27.690359    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ng7mg" event={"ID":"e51731de-fe8e-4ea7-a1f9-7661ebab8c1a","Type":"ContainerStarted","Data":"2098d13876206755cfe24fa68ed43b27133f0884c8f6d291ffed6c674bae2c62"}
Sep 09 20:17:27.690793 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:27.690487    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:27.709020 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:27.708972    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ng7mg" podStartSLOduration=17.467477002 podStartE2EDuration="19.708955734s" totalImagesPullingTime="2.241478732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:27.707910143 +0000 UTC m=+54.947012443" watchObservedRunningTime="2026-09-09 20:17:27.708955734 +0000 UTC m=+54.948058033"
Sep 09 20:17:31.622394 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.622362    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"]
Sep 09 20:17:31.625033 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.625005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"
Sep 09 20:17:31.625729 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.625706    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5"]
Sep 09 20:17:31.627240 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.627224    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5"
Sep 09 20:17:31.628619 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.628591    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:17:31.628619 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.628663    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:17:31.628619 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.628762    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 20:17:31.629076 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.628914    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:17:31.629563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.629544    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rvv8l\""
Sep 09 20:17:31.630179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.630161    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"]
Sep 09 20:17:31.630787 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.630771    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 20:17:31.632621 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.632377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.635899 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.635876    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 20:17:31.635999 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.635951    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 20:17:31.635999 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.635981    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 20:17:31.636143 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.636128    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 20:17:31.645273 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.645247    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"]
Sep 09 20:17:31.649209 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.649173    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5"]
Sep 09 20:17:31.650186 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.650165    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"]
Sep 09 20:17:31.657567 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.657547    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bb4794f65-7rk9b"]
Sep 09 20:17:31.711642 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.711612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-hub\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.711817 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.711666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.711817 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.711689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bf5d59d8-2swx5\" (UID: \"a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5"
Sep 09 20:17:31.711817 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.711750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.712058 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.711813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80afe0a1-17b4-4989-89ee-6646b0068b22-tmp\") pod \"klusterlet-addon-workmgr-69d9d9b74d-ccfbt\" (UID: \"80afe0a1-17b4-4989-89ee-6646b0068b22\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"
Sep 09 20:17:31.712058 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.711842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqh4d\" (UniqueName: \"kubernetes.io/projected/80afe0a1-17b4-4989-89ee-6646b0068b22-kube-api-access-rqh4d\") pod \"klusterlet-addon-workmgr-69d9d9b74d-ccfbt\" (UID: \"80afe0a1-17b4-4989-89ee-6646b0068b22\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"
Sep 09 20:17:31.712058 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.711872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-ca\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.712058 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.711931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.712058 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.711960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdqmj\" (UniqueName: \"kubernetes.io/projected/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-kube-api-access-cdqmj\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.712058 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.712017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/80afe0a1-17b4-4989-89ee-6646b0068b22-klusterlet-config\") pod \"klusterlet-addon-workmgr-69d9d9b74d-ccfbt\" (UID: \"80afe0a1-17b4-4989-89ee-6646b0068b22\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"
Sep 09 20:17:31.712326 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.712097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stz6x\" (UniqueName: \"kubernetes.io/projected/a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a-kube-api-access-stz6x\") pod \"managed-serviceaccount-addon-agent-bf5d59d8-2swx5\" (UID: \"a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5"
Sep 09 20:17:31.733932 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.733895    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84d69c5574-drmrj"]
Sep 09 20:17:31.738036 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.738003    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.764850 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.764819    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84d69c5574-drmrj"]
Sep 09 20:17:31.768986 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.768953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:31.812844 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.812813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stz6x\" (UniqueName: \"kubernetes.io/projected/a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a-kube-api-access-stz6x\") pod \"managed-serviceaccount-addon-agent-bf5d59d8-2swx5\" (UID: \"a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5"
Sep 09 20:17:31.813036 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.812852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-hub\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.813036 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.812884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.813036 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.812907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bf5d59d8-2swx5\" (UID: \"a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5"
Sep 09 20:17:31.813036 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.812935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.813036 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.812964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e592dada-173e-4c93-aa67-53e16ab500c7-registry-certificates\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.813036 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.812990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddws2\" (UniqueName: \"kubernetes.io/projected/e592dada-173e-4c93-aa67-53e16ab500c7-kube-api-access-ddws2\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.813036 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.813025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80afe0a1-17b4-4989-89ee-6646b0068b22-tmp\") pod \"klusterlet-addon-workmgr-69d9d9b74d-ccfbt\" (UID: \"80afe0a1-17b4-4989-89ee-6646b0068b22\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"
Sep 09 20:17:31.813881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.813081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqh4d\" (UniqueName: \"kubernetes.io/projected/80afe0a1-17b4-4989-89ee-6646b0068b22-kube-api-access-rqh4d\") pod \"klusterlet-addon-workmgr-69d9d9b74d-ccfbt\" (UID: \"80afe0a1-17b4-4989-89ee-6646b0068b22\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"
Sep 09 20:17:31.813881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.813124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e592dada-173e-4c93-aa67-53e16ab500c7-bound-sa-token\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.813881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.813154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-ca\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.813881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.813187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.813881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.813215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqmj\" (UniqueName: \"kubernetes.io/projected/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-kube-api-access-cdqmj\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.813881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.813249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e592dada-173e-4c93-aa67-53e16ab500c7-trusted-ca\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.813881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.813327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e592dada-173e-4c93-aa67-53e16ab500c7-ca-trust-extracted\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.813881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.813376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e592dada-173e-4c93-aa67-53e16ab500c7-image-registry-private-configuration\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.813881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.813394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80afe0a1-17b4-4989-89ee-6646b0068b22-tmp\") pod \"klusterlet-addon-workmgr-69d9d9b74d-ccfbt\" (UID: \"80afe0a1-17b4-4989-89ee-6646b0068b22\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"
Sep 09 20:17:31.813881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.813413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e592dada-173e-4c93-aa67-53e16ab500c7-registry-tls\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.813881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.813485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/80afe0a1-17b4-4989-89ee-6646b0068b22-klusterlet-config\") pod \"klusterlet-addon-workmgr-69d9d9b74d-ccfbt\" (UID: \"80afe0a1-17b4-4989-89ee-6646b0068b22\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"
Sep 09 20:17:31.813881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.813766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e592dada-173e-4c93-aa67-53e16ab500c7-installation-pull-secrets\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.814595 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.814042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.815767 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.815733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-hub\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.815887 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.815871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-ca\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.815964 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.815944    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bf5d59d8-2swx5\" (UID: \"a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5"
Sep 09 20:17:31.816092 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.816070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.816563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.816547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/80afe0a1-17b4-4989-89ee-6646b0068b22-klusterlet-config\") pod \"klusterlet-addon-workmgr-69d9d9b74d-ccfbt\" (UID: \"80afe0a1-17b4-4989-89ee-6646b0068b22\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"
Sep 09 20:17:31.816836 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.816819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.835597 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.835552    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stz6x\" (UniqueName: \"kubernetes.io/projected/a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a-kube-api-access-stz6x\") pod \"managed-serviceaccount-addon-agent-bf5d59d8-2swx5\" (UID: \"a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5"
Sep 09 20:17:31.843060 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.843029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqh4d\" (UniqueName: \"kubernetes.io/projected/80afe0a1-17b4-4989-89ee-6646b0068b22-kube-api-access-rqh4d\") pod \"klusterlet-addon-workmgr-69d9d9b74d-ccfbt\" (UID: \"80afe0a1-17b4-4989-89ee-6646b0068b22\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"
Sep 09 20:17:31.843159 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.843027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqmj\" (UniqueName: \"kubernetes.io/projected/65678c28-a1f3-4d63-9f07-5bfa1977bc6b-kube-api-access-cdqmj\") pod \"cluster-proxy-proxy-agent-69cfbfffd-vmpz9\" (UID: \"65678c28-a1f3-4d63-9f07-5bfa1977bc6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.906260 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.906184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:31.914349 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.914320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e592dada-173e-4c93-aa67-53e16ab500c7-registry-certificates\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.914458 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.914351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddws2\" (UniqueName: \"kubernetes.io/projected/e592dada-173e-4c93-aa67-53e16ab500c7-kube-api-access-ddws2\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.914458 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.914376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e592dada-173e-4c93-aa67-53e16ab500c7-bound-sa-token\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.914522 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.914478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e592dada-173e-4c93-aa67-53e16ab500c7-trusted-ca\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.914565 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.914521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e592dada-173e-4c93-aa67-53e16ab500c7-ca-trust-extracted\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.914643 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.914561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e592dada-173e-4c93-aa67-53e16ab500c7-image-registry-private-configuration\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.914643 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.914610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e592dada-173e-4c93-aa67-53e16ab500c7-registry-tls\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.914746 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.914679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e592dada-173e-4c93-aa67-53e16ab500c7-installation-pull-secrets\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.914922 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.914892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e592dada-173e-4c93-aa67-53e16ab500c7-ca-trust-extracted\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.915494 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.915469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e592dada-173e-4c93-aa67-53e16ab500c7-registry-certificates\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.915736 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.915710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e592dada-173e-4c93-aa67-53e16ab500c7-trusted-ca\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.917014 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.916974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e592dada-173e-4c93-aa67-53e16ab500c7-image-registry-private-configuration\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.917115 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.917097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e592dada-173e-4c93-aa67-53e16ab500c7-installation-pull-secrets\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.917216 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.917197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e592dada-173e-4c93-aa67-53e16ab500c7-registry-tls\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.923228 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.923207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddws2\" (UniqueName: \"kubernetes.io/projected/e592dada-173e-4c93-aa67-53e16ab500c7-kube-api-access-ddws2\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.923415 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.923395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e592dada-173e-4c93-aa67-53e16ab500c7-bound-sa-token\") pod \"image-registry-84d69c5574-drmrj\" (UID: \"e592dada-173e-4c93-aa67-53e16ab500c7\") " pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:31.935793 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.935768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"
Sep 09 20:17:31.951994 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.951965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5"
Sep 09 20:17:31.958102 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.958080    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"
Sep 09 20:17:31.995771 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:31.995741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:32.058900 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.058872    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:32.105133 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.105065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:32.193411 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.193331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:32.234199 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.234164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"]
Sep 09 20:17:32.237605 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:17:32.237561    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80afe0a1_17b4_4989_89ee_6646b0068b22.slice/crio-81c1c4396dd66409528e5436fb5b3a632da97f614299f131fa1bf37bb629d048 WatchSource:0}: Error finding container 81c1c4396dd66409528e5436fb5b3a632da97f614299f131fa1bf37bb629d048: Status 404 returned error can't find the container with id 81c1c4396dd66409528e5436fb5b3a632da97f614299f131fa1bf37bb629d048
Sep 09 20:17:32.261971 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.261943    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5"]
Sep 09 20:17:32.264826 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:17:32.264791    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda93a8c0d_6f8e_4a8b_ac84_0f1d6d34724a.slice/crio-0be81191acef7ca904b3c2af2be6d7c8ca08759bffd089ea04163af4bd4c6ada WatchSource:0}: Error finding container 0be81191acef7ca904b3c2af2be6d7c8ca08759bffd089ea04163af4bd4c6ada: Status 404 returned error can't find the container with id 0be81191acef7ca904b3c2af2be6d7c8ca08759bffd089ea04163af4bd4c6ada
Sep 09 20:17:32.288174 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.288141    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9"]
Sep 09 20:17:32.291052 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:17:32.291021    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65678c28_a1f3_4d63_9f07_5bfa1977bc6b.slice/crio-3a43f034e2f149c1fc125e2099e0fd5970c20aa8eadeae38d8f6e0993f098fe4 WatchSource:0}: Error finding container 3a43f034e2f149c1fc125e2099e0fd5970c20aa8eadeae38d8f6e0993f098fe4: Status 404 returned error can't find the container with id 3a43f034e2f149c1fc125e2099e0fd5970c20aa8eadeae38d8f6e0993f098fe4
Sep 09 20:17:32.369422 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.369385    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84d69c5574-drmrj"]
Sep 09 20:17:32.372594 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:17:32.372542    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode592dada_173e_4c93_aa67_53e16ab500c7.slice/crio-c1f26527abab4eba20798253f3c7ccfabc3c3500a07fefb3525fcda2c13e0b64 WatchSource:0}: Error finding container c1f26527abab4eba20798253f3c7ccfabc3c3500a07fefb3525fcda2c13e0b64: Status 404 returned error can't find the container with id c1f26527abab4eba20798253f3c7ccfabc3c3500a07fefb3525fcda2c13e0b64
Sep 09 20:17:32.705887 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.705852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84d69c5574-drmrj" event={"ID":"e592dada-173e-4c93-aa67-53e16ab500c7","Type":"ContainerStarted","Data":"eff5f1a14f5a764a714f49efb89ece4fd34a324547c9ea0cb0905052a2c9de10"}
Sep 09 20:17:32.706313 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.705910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84d69c5574-drmrj" event={"ID":"e592dada-173e-4c93-aa67-53e16ab500c7","Type":"ContainerStarted","Data":"c1f26527abab4eba20798253f3c7ccfabc3c3500a07fefb3525fcda2c13e0b64"}
Sep 09 20:17:32.706313 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.705978    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:32.707351 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.707325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5" event={"ID":"a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a","Type":"ContainerStarted","Data":"0be81191acef7ca904b3c2af2be6d7c8ca08759bffd089ea04163af4bd4c6ada"}
Sep 09 20:17:32.708510 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.708488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt" event={"ID":"80afe0a1-17b4-4989-89ee-6646b0068b22","Type":"ContainerStarted","Data":"81c1c4396dd66409528e5436fb5b3a632da97f614299f131fa1bf37bb629d048"}
Sep 09 20:17:32.709686 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.709666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9" event={"ID":"65678c28-a1f3-4d63-9f07-5bfa1977bc6b","Type":"ContainerStarted","Data":"3a43f034e2f149c1fc125e2099e0fd5970c20aa8eadeae38d8f6e0993f098fe4"}
Sep 09 20:17:32.733300 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.733198    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84d69c5574-drmrj" podStartSLOduration=1.733181365 podStartE2EDuration="1.733181365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:32.732475507 +0000 UTC m=+59.971577807" watchObservedRunningTime="2026-09-09 20:17:32.733181365 +0000 UTC m=+59.972283662"
Sep 09 20:17:32.753392 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.753357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:32.827023 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:32.826987    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:33.583843 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:33.583802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:33.678407 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:33.677853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:33.751435 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:33.751400    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:33.816801 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:33.816762    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:34.620624 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:34.620548    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hcfl"
Sep 09 20:17:34.781104 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:34.781023    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:34.882619 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:34.882511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:35.624666 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:35.624622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:35.692561 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:35.692525    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:37.695434 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:37.695404    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ng7mg"
Sep 09 20:17:37.701130 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:37.701100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:37.785662 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:37.785603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:38.054414 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:38.054377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:38.979053 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:38.979020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs\") pod \"network-metrics-daemon-86bgz\" (UID: \"7377be15-1184-4239-a696-75be3f1d835d\") " pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:17:38.982496 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:38.982470    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:17:38.992312 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:38.992286    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7377be15-1184-4239-a696-75be3f1d835d-metrics-certs\") pod \"network-metrics-daemon-86bgz\" (UID: \"7377be15-1184-4239-a696-75be3f1d835d\") " pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:17:39.080480 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.080405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:17:39.082966 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.082944    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:17:39.093531 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.093504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d65fd73a-56b3-4e21-988d-cd2d26bb8847-original-pull-secret\") pod \"global-pull-secret-syncer-lsr8k\" (UID: \"d65fd73a-56b3-4e21-988d-cd2d26bb8847\") " pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:17:39.181089 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.181046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfsrl\" (UniqueName: \"kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl\") pod \"network-check-target-pkqbv\" (UID: \"011ed84f-e0dd-495c-9eb3-fd0ad001c3d5\") " pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:17:39.183633 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.183608    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:17:39.194027 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.193997    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:17:39.204274 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.204241    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfsrl\" (UniqueName: \"kubernetes.io/projected/011ed84f-e0dd-495c-9eb3-fd0ad001c3d5-kube-api-access-cfsrl\") pod \"network-check-target-pkqbv\" (UID: \"011ed84f-e0dd-495c-9eb3-fd0ad001c3d5\") " pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:17:39.220456 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.220429    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c88hn\""
Sep 09 20:17:39.226749 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.226721    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wv6m4\""
Sep 09 20:17:39.228595 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.228561    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:17:39.235428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.235409    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bgz"
Sep 09 20:17:39.238670 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.238651    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lsr8k"
Sep 09 20:17:39.509332 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.509301    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pkqbv"]
Sep 09 20:17:39.512619 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:17:39.512592    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011ed84f_e0dd_495c_9eb3_fd0ad001c3d5.slice/crio-a501b974f21ca5cd1da57a7eac8eb4a7111b0b362fb5859d5cbcd1dd395d521e WatchSource:0}: Error finding container a501b974f21ca5cd1da57a7eac8eb4a7111b0b362fb5859d5cbcd1dd395d521e: Status 404 returned error can't find the container with id a501b974f21ca5cd1da57a7eac8eb4a7111b0b362fb5859d5cbcd1dd395d521e
Sep 09 20:17:39.538987 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.538955    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-86bgz"]
Sep 09 20:17:39.541919 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:17:39.541887    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7377be15_1184_4239_a696_75be3f1d835d.slice/crio-c27b8189d9d76793a4337c05ea76ac5462691da3ff105886e3bf3c549316b89a WatchSource:0}: Error finding container c27b8189d9d76793a4337c05ea76ac5462691da3ff105886e3bf3c549316b89a: Status 404 returned error can't find the container with id c27b8189d9d76793a4337c05ea76ac5462691da3ff105886e3bf3c549316b89a
Sep 09 20:17:39.563893 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.563861    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lsr8k"]
Sep 09 20:17:39.567202 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:17:39.567171    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65fd73a_56b3_4e21_988d_cd2d26bb8847.slice/crio-18dbd91506a3204e6d235ab9c925fc429c162c33647424fc52c4b56497af67b4 WatchSource:0}: Error finding container 18dbd91506a3204e6d235ab9c925fc429c162c33647424fc52c4b56497af67b4: Status 404 returned error can't find the container with id 18dbd91506a3204e6d235ab9c925fc429c162c33647424fc52c4b56497af67b4
Sep 09 20:17:39.748962 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.748924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5" event={"ID":"a93a8c0d-6f8e-4a8b-ac84-0f1d6d34724a","Type":"ContainerStarted","Data":"3e5474b7fbf33516734055ecbdb3bb377de388a29900eedf852adc9d67a573fd"}
Sep 09 20:17:39.750179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.750151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pkqbv" event={"ID":"011ed84f-e0dd-495c-9eb3-fd0ad001c3d5","Type":"ContainerStarted","Data":"a501b974f21ca5cd1da57a7eac8eb4a7111b0b362fb5859d5cbcd1dd395d521e"}
Sep 09 20:17:39.751610 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.751565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt" event={"ID":"80afe0a1-17b4-4989-89ee-6646b0068b22","Type":"ContainerStarted","Data":"004deefc6113fd05171ebe6a4f802341a6f90656d39d805b2e69830b5f8a5b22"}
Sep 09 20:17:39.751787 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.751767    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"
Sep 09 20:17:39.753842 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.753679    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt"
Sep 09 20:17:39.756179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.756155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9" event={"ID":"65678c28-a1f3-4d63-9f07-5bfa1977bc6b","Type":"ContainerStarted","Data":"c157b44afd497e8cac46a63419a43cb5444cec0d8017d7eb3acfad579417a854"}
Sep 09 20:17:39.756284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.756190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9" event={"ID":"65678c28-a1f3-4d63-9f07-5bfa1977bc6b","Type":"ContainerStarted","Data":"29c9fa6007512b8167b006d9b1dc9591c7db27f7c4c76815302d4d58f942569e"}
Sep 09 20:17:39.756284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.756204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9" event={"ID":"65678c28-a1f3-4d63-9f07-5bfa1977bc6b","Type":"ContainerStarted","Data":"847b4ca40a40af699e8355efe96bc5679f9362ee5b34250e86e14168cce6b66e"}
Sep 09 20:17:39.757614 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.757592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lsr8k" event={"ID":"d65fd73a-56b3-4e21-988d-cd2d26bb8847","Type":"ContainerStarted","Data":"18dbd91506a3204e6d235ab9c925fc429c162c33647424fc52c4b56497af67b4"}
Sep 09 20:17:39.758598 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.758565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-86bgz" event={"ID":"7377be15-1184-4239-a696-75be3f1d835d","Type":"ContainerStarted","Data":"c27b8189d9d76793a4337c05ea76ac5462691da3ff105886e3bf3c549316b89a"}
Sep 09 20:17:39.766323 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.766284    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bf5d59d8-2swx5" podStartSLOduration=2.189801754 podStartE2EDuration="8.7662715s" totalImagesPullingTime="6.576469746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:39.765674663 +0000 UTC m=+67.004776965" watchObservedRunningTime="2026-09-09 20:17:39.7662715 +0000 UTC m=+67.005373800"
Sep 09 20:17:39.787222 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.787170    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69cfbfffd-vmpz9" podStartSLOduration=2.222112917 podStartE2EDuration="8.787148916s" totalImagesPullingTime="6.565035999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:39.786781163 +0000 UTC m=+67.025883462" watchObservedRunningTime="2026-09-09 20:17:39.787148916 +0000 UTC m=+67.026251214"
Sep 09 20:17:39.803607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.803540    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69d9d9b74d-ccfbt" podStartSLOduration=2.182378265 podStartE2EDuration="8.803522601s" totalImagesPullingTime="6.621144336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:39.802651064 +0000 UTC m=+67.041753363" watchObservedRunningTime="2026-09-09 20:17:39.803522601 +0000 UTC m=+67.042624903"
Sep 09 20:17:39.989699 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.989656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:39.992444 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:39.992415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d384213-5569-4654-bb4f-6e1834675c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dcf7r\" (UID: \"4d384213-5569-4654-bb4f-6e1834675c73\") " pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:40.236737 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:40.236538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s82pj\""
Sep 09 20:17:40.244992 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:40.244959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dcf7r"
Sep 09 20:17:40.462636 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:40.462597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:40.901988 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:40.901898    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dcf7r"]
Sep 09 20:17:40.906455 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:17:40.906418    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d384213_5569_4654_bb4f_6e1834675c73.slice/crio-80928130dba390aed2085937aae16162c88c52178fd197a288d1a683c714d085 WatchSource:0}: Error finding container 80928130dba390aed2085937aae16162c88c52178fd197a288d1a683c714d085: Status 404 returned error can't find the container with id 80928130dba390aed2085937aae16162c88c52178fd197a288d1a683c714d085
Sep 09 20:17:41.655792 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:41.655759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:41.665426 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:41.665378    2563 patch_prober.go:28] interesting pod/image-registry-7bb4794f65-7rk9b container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 20:17:41.665567 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:41.665436    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b" podUID="519bb8de-4eec-41ca-b66c-122e0bf314ea" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:17:41.740713 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:41.740671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:41.771310 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:41.771264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dcf7r" event={"ID":"4d384213-5569-4654-bb4f-6e1834675c73","Type":"ContainerStarted","Data":"55abc1336c47a41f32b69dd558fc2145fbda15a4218b0d3d5a1c2aa401b4060b"}
Sep 09 20:17:41.771464 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:41.771339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dcf7r" event={"ID":"4d384213-5569-4654-bb4f-6e1834675c73","Type":"ContainerStarted","Data":"8ca82cada552c8a5c307d16de8f1a3bfda238c24912b74eb37f430a9ade72a5f"}
Sep 09 20:17:41.771464 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:41.771366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dcf7r" event={"ID":"4d384213-5569-4654-bb4f-6e1834675c73","Type":"ContainerStarted","Data":"80928130dba390aed2085937aae16162c88c52178fd197a288d1a683c714d085"}
Sep 09 20:17:41.773335 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:41.773298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-86bgz" event={"ID":"7377be15-1184-4239-a696-75be3f1d835d","Type":"ContainerStarted","Data":"e5ef1d50d0e5ccd824c437151aaa905299fada89a901a18730c56d2ee2cae2e8"}
Sep 09 20:17:41.773465 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:41.773352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-86bgz" event={"ID":"7377be15-1184-4239-a696-75be3f1d835d","Type":"ContainerStarted","Data":"979c228c3daf27dcc5ee777135e655cab805ba890c3b7e20a7c72e64376c86f1"}
Sep 09 20:17:41.790897 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:41.790832    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-86bgz" podStartSLOduration=66.492292526 podStartE2EDuration="1m7.790810664s" totalImagesPullingTime="1.298518138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:41.789138696 +0000 UTC m=+69.028240996" watchObservedRunningTime="2026-09-09 20:17:41.790810664 +0000 UTC m=+69.029912966"
Sep 09 20:17:41.948184 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:41.948095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:42.036615 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.036558    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:42.146120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.146037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:42.217686 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.217178    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d7sfj"]
Sep 09 20:17:42.221106 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.221076    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.224683 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.224658    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:17:42.224887 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.224869    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:17:42.225046 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.225029    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:17:42.225275 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.225259    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lpcq2\""
Sep 09 20:17:42.225607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.225592    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:17:42.225760 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.225745    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:17:42.225950 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.225936    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:17:42.255027 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.254943    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:42.311129 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.310896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-metrics-client-ca\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.311129 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.310945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-wtmp\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.311129 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.310975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.311129 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.310996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.311129 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.311032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xfk4\" (UniqueName: \"kubernetes.io/projected/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-kube-api-access-5xfk4\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.311129 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.311058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-tls\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.311129 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.311089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-textfile\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.311129 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.311123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-sys\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.311129 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.311137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-root\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.412369 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.412327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xfk4\" (UniqueName: \"kubernetes.io/projected/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-kube-api-access-5xfk4\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.412563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.412387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-tls\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.412563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.412426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-textfile\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.412563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.412451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-sys\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.412563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.412474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-root\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.412563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.412520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-metrics-client-ca\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.412563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.412546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-wtmp\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.412928 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.412599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.412928 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.412630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.413319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.413292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.414059 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.414036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-root\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.415604 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.414270    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-textfile\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.415604 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.414320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-sys\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.415604 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.414434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-wtmp\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.415604 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.414857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-metrics-client-ca\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.417785 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.417761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.419231 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.419208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-node-exporter-tls\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.423542 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.423517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xfk4\" (UniqueName: \"kubernetes.io/projected/1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98-kube-api-access-5xfk4\") pod \"node-exporter-d7sfj\" (UID: \"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98\") " pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.510304 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.510168    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:42.537233 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.537188    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d7sfj"
Sep 09 20:17:42.610749 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.610705    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:42.712086 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.712053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:42.814128 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.814025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:42.910083 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:42.910045    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:43.010924 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:43.010851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:43.115094 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:43.115012    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:43.217086 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:43.217047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:43.298478 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:43.298441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:43.425464 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:43.425341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:43.525965 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:43.525931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:43.717299 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:43.717197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:44.535092 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:17:44.535050    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8fb8cd_6fbe_4b8a_bce0_8c27675afd98.slice/crio-552b49a1e5f7025c1720fc8b075f03f0f2f1aef511b967fbc4518a6c5a93a191 WatchSource:0}: Error finding container 552b49a1e5f7025c1720fc8b075f03f0f2f1aef511b967fbc4518a6c5a93a191: Status 404 returned error can't find the container with id 552b49a1e5f7025c1720fc8b075f03f0f2f1aef511b967fbc4518a6c5a93a191
Sep 09 20:17:44.787850 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:44.787759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lsr8k" event={"ID":"d65fd73a-56b3-4e21-988d-cd2d26bb8847","Type":"ContainerStarted","Data":"b6a772480c82d7d87f109c68d33e52a3042fce0c1a1176e864b779e6051bc90d"}
Sep 09 20:17:44.790262 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:44.790229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pkqbv" event={"ID":"011ed84f-e0dd-495c-9eb3-fd0ad001c3d5","Type":"ContainerStarted","Data":"9195a8cd7280b573161dcddff16162e0a553988476e3792b2bc697e6b2f7eed9"}
Sep 09 20:17:44.790399 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:44.790309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:17:44.792701 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:44.792669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7sfj" event={"ID":"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98","Type":"ContainerStarted","Data":"552b49a1e5f7025c1720fc8b075f03f0f2f1aef511b967fbc4518a6c5a93a191"}
Sep 09 20:17:44.805638 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:44.805552    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lsr8k" podStartSLOduration=65.822919327 podStartE2EDuration="1m10.805530064s" totalImagesPullingTime="4.982610737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:44.804844908 +0000 UTC m=+72.043947209" watchObservedRunningTime="2026-09-09 20:17:44.805530064 +0000 UTC m=+72.044632368"
Sep 09 20:17:44.831190 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:44.831120    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pkqbv" podStartSLOduration=65.806649739 podStartE2EDuration="1m10.831097404s" totalImagesPullingTime="5.024447665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:44.82928756 +0000 UTC m=+72.068389862" watchObservedRunningTime="2026-09-09 20:17:44.831097404 +0000 UTC m=+72.070199703"
Sep 09 20:17:44.843678 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:44.843642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:44.929441 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:44.929404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:45.339890 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:45.339848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:45.417949 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:45.417910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:45.632994 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:45.632911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:45.798269 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:45.798233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dcf7r" event={"ID":"4d384213-5569-4654-bb4f-6e1834675c73","Type":"ContainerStarted","Data":"d9d7ca8f39892462b84b2bc9883e8adf2d818882c83ddfb876e40c9405669018"}
Sep 09 20:17:45.799835 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:45.799801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7sfj" event={"ID":"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98","Type":"ContainerStarted","Data":"b0b442a852547825b3a3144c5a83e100f020b3059fee03757929e2eb32c710e6"}
Sep 09 20:17:45.821164 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:45.821100    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dcf7r" podStartSLOduration=33.179575691 podStartE2EDuration="37.821078115s" totalImagesPullingTime="4.641502424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:17:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:45.820651951 +0000 UTC m=+73.059754252" watchObservedRunningTime="2026-09-09 20:17:45.821078115 +0000 UTC m=+73.060180416"
Sep 09 20:17:45.864881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:45.864850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:46.804736 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:46.804704    2563 generic.go:356] "Generic (PLEG): container finished" podID="1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98" containerID="b0b442a852547825b3a3144c5a83e100f020b3059fee03757929e2eb32c710e6" exitCode=0
Sep 09 20:17:46.805162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:46.804788    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7sfj" event={"ID":"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98","Type":"ContainerDied","Data":"b0b442a852547825b3a3144c5a83e100f020b3059fee03757929e2eb32c710e6"}
Sep 09 20:17:46.855220 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:46.855182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:46.948803 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:46.948766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:47.058300 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:47.058262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:47.157758 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:47.157713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:47.810694 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:47.810654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7sfj" event={"ID":"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98","Type":"ContainerStarted","Data":"97ac786c08b453669dc9e821d4a111e5a4076f6e2af47e5ad9d2437c8556d628"}
Sep 09 20:17:47.811074 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:47.810707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7sfj" event={"ID":"1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98","Type":"ContainerStarted","Data":"066708eb558a1b76bb2d052f8825fb84179f796c925e8b5e71b2c5d3fba1f0c9"}
Sep 09 20:17:47.834947 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:47.834895    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d7sfj" podStartSLOduration=4.605213728 podStartE2EDuration="5.834879607s" totalImagesPullingTime="1.163424436s" totalInitContainerRuntime="66.241443ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:47.832909977 +0000 UTC m=+75.072012272" watchObservedRunningTime="2026-09-09 20:17:47.834879607 +0000 UTC m=+75.073981906"
Sep 09 20:17:48.627190 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:48.627142    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:48.715334 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:48.715299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:51.598003 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:51.597968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:51.661877 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:51.661849    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:51.688829 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:51.688791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:52.337760 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:52.337726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:52.402045 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:52.402005    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:53.723052 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:53.723024    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84d69c5574-drmrj"
Sep 09 20:17:53.908270 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:53.908230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:54.006175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:54.006067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:55.349534 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:55.349404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:55.457287 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:55.457253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:55.837447 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:55.837413    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:55.955688 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:55.955639    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:56.054989 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:56.054954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:56.623014 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:56.622977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:56.676096 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:56.676051    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b" podUID="519bb8de-4eec-41ca-b66c-122e0bf314ea" containerName="registry" containerID="cri-o://ca5811ecb2df0b4c570731e15c58980e36f2b59780c22ae1f813169d230ec7d1" gracePeriod=30
Sep 09 20:17:56.724367 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:56.724331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:56.844320 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:56.844290    2563 generic.go:356] "Generic (PLEG): container finished" podID="519bb8de-4eec-41ca-b66c-122e0bf314ea" containerID="ca5811ecb2df0b4c570731e15c58980e36f2b59780c22ae1f813169d230ec7d1" exitCode=0
Sep 09 20:17:56.844443 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:56.844369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b" event={"ID":"519bb8de-4eec-41ca-b66c-122e0bf314ea","Type":"ContainerDied","Data":"ca5811ecb2df0b4c570731e15c58980e36f2b59780c22ae1f813169d230ec7d1"}
Sep 09 20:17:56.911122 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:56.911093    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:57.035186 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.035146    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-certificates\") pod \"519bb8de-4eec-41ca-b66c-122e0bf314ea\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") "
Sep 09 20:17:57.035378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.035194    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/519bb8de-4eec-41ca-b66c-122e0bf314ea-trusted-ca\") pod \"519bb8de-4eec-41ca-b66c-122e0bf314ea\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") "
Sep 09 20:17:57.035378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.035220    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-bound-sa-token\") pod \"519bb8de-4eec-41ca-b66c-122e0bf314ea\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") "
Sep 09 20:17:57.035378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.035238    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/519bb8de-4eec-41ca-b66c-122e0bf314ea-installation-pull-secrets\") pod \"519bb8de-4eec-41ca-b66c-122e0bf314ea\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") "
Sep 09 20:17:57.035378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.035254    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/519bb8de-4eec-41ca-b66c-122e0bf314ea-image-registry-private-configuration\") pod \"519bb8de-4eec-41ca-b66c-122e0bf314ea\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") "
Sep 09 20:17:57.035378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.035296    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/519bb8de-4eec-41ca-b66c-122e0bf314ea-ca-trust-extracted\") pod \"519bb8de-4eec-41ca-b66c-122e0bf314ea\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") "
Sep 09 20:17:57.035378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.035324    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vgqc\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-kube-api-access-9vgqc\") pod \"519bb8de-4eec-41ca-b66c-122e0bf314ea\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") "
Sep 09 20:17:57.035378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.035357    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls\") pod \"519bb8de-4eec-41ca-b66c-122e0bf314ea\" (UID: \"519bb8de-4eec-41ca-b66c-122e0bf314ea\") "
Sep 09 20:17:57.035763 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.035732    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "519bb8de-4eec-41ca-b66c-122e0bf314ea" (UID: "519bb8de-4eec-41ca-b66c-122e0bf314ea"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:57.035876 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.035846    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519bb8de-4eec-41ca-b66c-122e0bf314ea-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "519bb8de-4eec-41ca-b66c-122e0bf314ea" (UID: "519bb8de-4eec-41ca-b66c-122e0bf314ea"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:57.038276 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.038237    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519bb8de-4eec-41ca-b66c-122e0bf314ea-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "519bb8de-4eec-41ca-b66c-122e0bf314ea" (UID: "519bb8de-4eec-41ca-b66c-122e0bf314ea"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:17:57.038377 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.038326    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519bb8de-4eec-41ca-b66c-122e0bf314ea-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "519bb8de-4eec-41ca-b66c-122e0bf314ea" (UID: "519bb8de-4eec-41ca-b66c-122e0bf314ea"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:17:57.038377 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.038325    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-kube-api-access-9vgqc" (OuterVolumeSpecName: "kube-api-access-9vgqc") pod "519bb8de-4eec-41ca-b66c-122e0bf314ea" (UID: "519bb8de-4eec-41ca-b66c-122e0bf314ea"). InnerVolumeSpecName "kube-api-access-9vgqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:17:57.038473 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.038371    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "519bb8de-4eec-41ca-b66c-122e0bf314ea" (UID: "519bb8de-4eec-41ca-b66c-122e0bf314ea"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:17:57.038473 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.038436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "519bb8de-4eec-41ca-b66c-122e0bf314ea" (UID: "519bb8de-4eec-41ca-b66c-122e0bf314ea"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:17:57.044445 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.044420    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519bb8de-4eec-41ca-b66c-122e0bf314ea-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "519bb8de-4eec-41ca-b66c-122e0bf314ea" (UID: "519bb8de-4eec-41ca-b66c-122e0bf314ea"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:17:57.136503 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.136469    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/519bb8de-4eec-41ca-b66c-122e0bf314ea-installation-pull-secrets\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:17:57.136503 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.136500    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/519bb8de-4eec-41ca-b66c-122e0bf314ea-image-registry-private-configuration\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:17:57.136503 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.136511    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/519bb8de-4eec-41ca-b66c-122e0bf314ea-ca-trust-extracted\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:17:57.136745 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.136520    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vgqc\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-kube-api-access-9vgqc\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:17:57.136745 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.136529    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:17:57.136745 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.136537    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/519bb8de-4eec-41ca-b66c-122e0bf314ea-registry-certificates\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:17:57.136745 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.136546    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/519bb8de-4eec-41ca-b66c-122e0bf314ea-trusted-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:17:57.136745 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.136557    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/519bb8de-4eec-41ca-b66c-122e0bf314ea-bound-sa-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:17:57.849213 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.849177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b" event={"ID":"519bb8de-4eec-41ca-b66c-122e0bf314ea","Type":"ContainerDied","Data":"9895e52eeb4c7536880e34c94e8eb5c8bb11579b272b448116c79bb008633bbe"}
Sep 09 20:17:57.849213 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.849206    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bb4794f65-7rk9b"
Sep 09 20:17:57.849705 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.849259    2563 scope.go:117] "RemoveContainer" containerID="ca5811ecb2df0b4c570731e15c58980e36f2b59780c22ae1f813169d230ec7d1"
Sep 09 20:17:57.888898 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.888860    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bb4794f65-7rk9b"]
Sep 09 20:17:57.901348 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:57.901317    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bb4794f65-7rk9b"]
Sep 09 20:17:59.312211 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:17:59.312170    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="519bb8de-4eec-41ca-b66c-122e0bf314ea" path="/var/lib/kubelet/pods/519bb8de-4eec-41ca-b66c-122e0bf314ea/volumes"
Sep 09 20:18:00.340793 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:00.340760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:00.446969 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:00.446928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:01.153157 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:01.153119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:01.249995 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:01.249953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:04.910636 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:04.910597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:05.005017 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:05.004981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:05.130912 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:05.130872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:05.731404 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:05.731363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:05.832126 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:05.832088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:10.784331 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:10.784290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:10.849748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:10.849703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:13.344647 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:13.344607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:15.802734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:15.802703    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pkqbv"
Sep 09 20:18:19.947331 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:19.947297    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:21.888878 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:21.888842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:21.985068 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:21.985010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:25.799731 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:25.799692    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:25.902698 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:25.902658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:31.100858 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:18:31.100816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:01.577031 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:19:01.576986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:01.858424 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:19:01.858327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:10.357192 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:19:10.357157    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:10.455037 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:19:10.454990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:14.101676 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:19:14.101643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:21.014502 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:19:21.014459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:21.231985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:19:21.231948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:21.298132 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:19:21.298051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:23.643456 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:19:23.643418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:51.484838 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:19:51.484801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:51.582385 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:19:51.582348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:16.946521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:16.946485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:17.109249 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:17.109194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:19.399349 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:19.399310    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:21.962365 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:21.962332    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:22.064040 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:22.064006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:22.140645 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:22.140612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:22.744453 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:22.744419    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:22.812010 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:22.811826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:28.861959 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:28.861925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:28.940012 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:28.939977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:33.279353 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:33.279319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:33.379183 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:33.379149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:33.438376 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:33.438329    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:40.787181 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:40.787101    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:40.888314 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:20:40.888280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:20.947308 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:21:20.947275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:22.742051 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:21:22.742012    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:33.156672 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:21:33.156643    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:21:47.977518 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:21:47.977474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:48.994745 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:21:48.994709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:51.795086 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:21:51.795051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:51.902814 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:21:51.902779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:53.920810 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:21:53.920774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:54.042074 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:21:54.042020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:56.295590 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:21:56.295534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:56.695567 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:21:56.695489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:58.780116 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:21:58.780083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:58.881393 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:21:58.881361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:04.513513 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:04.513479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:06.513196 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:06.513151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:06.611197 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:06.611165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:13.308768 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:13.308725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:13.410688 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:13.410651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:15.000526 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:15.000490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:21.816795 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:21.816753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:21.917952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:21.917915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:26.597586 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:26.597485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:26.691149 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:26.691113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:27.208611 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.208565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:27.467874 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.467782    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:27.580225 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.580186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:27.625561 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.625525    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"]
Sep 09 20:22:27.625927 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.625775    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="519bb8de-4eec-41ca-b66c-122e0bf314ea" containerName="registry"
Sep 09 20:22:27.625927 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.625787    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="519bb8de-4eec-41ca-b66c-122e0bf314ea" containerName="registry"
Sep 09 20:22:27.625927 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.625837    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="519bb8de-4eec-41ca-b66c-122e0bf314ea" containerName="registry"
Sep 09 20:22:27.628085 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.628064    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.630818 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.630773    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 20:22:27.630953 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.630941    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 20:22:27.631171 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.631151    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 20:22:27.631264 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.631151    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p75md\""
Sep 09 20:22:27.631684 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.631669    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 20:22:27.660888 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.660857    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"]
Sep 09 20:22:27.674588 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.674547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db07ac0a-97c1-415a-b685-37f8e342af6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.674718 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.674603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db07ac0a-97c1-415a-b685-37f8e342af6f-manifests\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.674718 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.674663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db07ac0a-97c1-415a-b685-37f8e342af6f-tmp\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.674718 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.674680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db07ac0a-97c1-415a-b685-37f8e342af6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.674817 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.674735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s87x4\" (UniqueName: \"kubernetes.io/projected/db07ac0a-97c1-415a-b685-37f8e342af6f-kube-api-access-s87x4\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.776111 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.776026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db07ac0a-97c1-415a-b685-37f8e342af6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.776111 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.776069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db07ac0a-97c1-415a-b685-37f8e342af6f-manifests\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.776111 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.776112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db07ac0a-97c1-415a-b685-37f8e342af6f-tmp\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.776375 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.776207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db07ac0a-97c1-415a-b685-37f8e342af6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.776375 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.776271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s87x4\" (UniqueName: \"kubernetes.io/projected/db07ac0a-97c1-415a-b685-37f8e342af6f-kube-api-access-s87x4\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.776506 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.776486    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db07ac0a-97c1-415a-b685-37f8e342af6f-manifests\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.778353 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.778331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db07ac0a-97c1-415a-b685-37f8e342af6f-tmp\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.778584 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.778553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db07ac0a-97c1-415a-b685-37f8e342af6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.778727 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.778707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db07ac0a-97c1-415a-b685-37f8e342af6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.823499 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.823458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s87x4\" (UniqueName: \"kubernetes.io/projected/db07ac0a-97c1-415a-b685-37f8e342af6f-kube-api-access-s87x4\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-7n27t\" (UID: \"db07ac0a-97c1-415a-b685-37f8e342af6f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.869089 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.869051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:27.938402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.938350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:27.966868 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:27.966835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:28.070332 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:28.070235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:28.226597 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:28.226539    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"]
Sep 09 20:22:28.229523 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:22:28.229492    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb07ac0a_97c1_415a_b685_37f8e342af6f.slice/crio-247cf3a311baab3feba3f18f70f1efbc6412acda9c1de21fbbc8b33c4627c66c WatchSource:0}: Error finding container 247cf3a311baab3feba3f18f70f1efbc6412acda9c1de21fbbc8b33c4627c66c: Status 404 returned error can't find the container with id 247cf3a311baab3feba3f18f70f1efbc6412acda9c1de21fbbc8b33c4627c66c
Sep 09 20:22:28.231362 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:28.231341    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:22:28.296806 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:28.296772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:28.396534 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:28.396439    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:28.718194 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:28.718151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t" event={"ID":"db07ac0a-97c1-415a-b685-37f8e342af6f","Type":"ContainerStarted","Data":"247cf3a311baab3feba3f18f70f1efbc6412acda9c1de21fbbc8b33c4627c66c"}
Sep 09 20:22:31.311884 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:31.311846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:31.410884 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:31.410838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:31.730924 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:31.730889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t" event={"ID":"db07ac0a-97c1-415a-b685-37f8e342af6f","Type":"ContainerStarted","Data":"b855da1526c7fe188c6ff05e848064e1888c8232bbeb581676dec1cf6b9ce26e"}
Sep 09 20:22:32.735676 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:32.735639    2563 generic.go:356] "Generic (PLEG): container finished" podID="db07ac0a-97c1-415a-b685-37f8e342af6f" containerID="b855da1526c7fe188c6ff05e848064e1888c8232bbeb581676dec1cf6b9ce26e" exitCode=0
Sep 09 20:22:32.736131 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:32.735692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t" event={"ID":"db07ac0a-97c1-415a-b685-37f8e342af6f","Type":"ContainerDied","Data":"b855da1526c7fe188c6ff05e848064e1888c8232bbeb581676dec1cf6b9ce26e"}
Sep 09 20:22:33.740521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:33.740486    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t" event={"ID":"db07ac0a-97c1-415a-b685-37f8e342af6f","Type":"ContainerStarted","Data":"0ef1c415f115140547d6f3666bcb4f2a966b6c3f11bdaf296aeff06899583179"}
Sep 09 20:22:33.740883 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:33.740621    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:33.763944 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:33.763897    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t" podStartSLOduration=2.361475791 podStartE2EDuration="6.763880457s" totalImagesPullingTime="3.384629996s" totalInitContainerRuntime="1.01777467s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:33.763540604 +0000 UTC m=+361.002642907" watchObservedRunningTime="2026-09-09 20:22:33.763880457 +0000 UTC m=+361.002982757"
Sep 09 20:22:44.746917 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:44.746877    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-7n27t"
Sep 09 20:22:46.544227 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:46.544183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:46.685278 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:46.685245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:47.002198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.002161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:47.116916 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.116884    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"]
Sep 09 20:22:47.126938 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.126912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"
Sep 09 20:22:47.142651 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.142623    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 20:22:47.142785 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.142740    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 20:22:47.143655 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.143636    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fhwmj\""
Sep 09 20:22:47.143731 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.143652    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 20:22:47.143731 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.143713    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 20:22:47.149519 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.149494    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"]
Sep 09 20:22:47.212376 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.212335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03d79687-ff1c-49fb-ae29-37e8901a5e68-tmp\") pod \"kube-auth-proxy-758cfc744b-w7r8z\" (UID: \"03d79687-ff1c-49fb-ae29-37e8901a5e68\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"
Sep 09 20:22:47.212601 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.212384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7tqw\" (UniqueName: \"kubernetes.io/projected/03d79687-ff1c-49fb-ae29-37e8901a5e68-kube-api-access-h7tqw\") pod \"kube-auth-proxy-758cfc744b-w7r8z\" (UID: \"03d79687-ff1c-49fb-ae29-37e8901a5e68\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"
Sep 09 20:22:47.212601 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.212475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03d79687-ff1c-49fb-ae29-37e8901a5e68-tls-certs\") pod \"kube-auth-proxy-758cfc744b-w7r8z\" (UID: \"03d79687-ff1c-49fb-ae29-37e8901a5e68\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"
Sep 09 20:22:47.313679 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.313590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03d79687-ff1c-49fb-ae29-37e8901a5e68-tls-certs\") pod \"kube-auth-proxy-758cfc744b-w7r8z\" (UID: \"03d79687-ff1c-49fb-ae29-37e8901a5e68\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"
Sep 09 20:22:47.313679 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.313669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03d79687-ff1c-49fb-ae29-37e8901a5e68-tmp\") pod \"kube-auth-proxy-758cfc744b-w7r8z\" (UID: \"03d79687-ff1c-49fb-ae29-37e8901a5e68\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"
Sep 09 20:22:47.313882 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.313714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7tqw\" (UniqueName: \"kubernetes.io/projected/03d79687-ff1c-49fb-ae29-37e8901a5e68-kube-api-access-h7tqw\") pod \"kube-auth-proxy-758cfc744b-w7r8z\" (UID: \"03d79687-ff1c-49fb-ae29-37e8901a5e68\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"
Sep 09 20:22:47.315952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.315927    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03d79687-ff1c-49fb-ae29-37e8901a5e68-tmp\") pod \"kube-auth-proxy-758cfc744b-w7r8z\" (UID: \"03d79687-ff1c-49fb-ae29-37e8901a5e68\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"
Sep 09 20:22:47.316279 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.316256    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03d79687-ff1c-49fb-ae29-37e8901a5e68-tls-certs\") pod \"kube-auth-proxy-758cfc744b-w7r8z\" (UID: \"03d79687-ff1c-49fb-ae29-37e8901a5e68\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"
Sep 09 20:22:47.332457 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.332430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7tqw\" (UniqueName: \"kubernetes.io/projected/03d79687-ff1c-49fb-ae29-37e8901a5e68-kube-api-access-h7tqw\") pod \"kube-auth-proxy-758cfc744b-w7r8z\" (UID: \"03d79687-ff1c-49fb-ae29-37e8901a5e68\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"
Sep 09 20:22:47.421419 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.421379    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:47.435918 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.435890    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"
Sep 09 20:22:47.733528 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.733501    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z"]
Sep 09 20:22:47.736935 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:22:47.736911    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d79687_ff1c_49fb_ae29_37e8901a5e68.slice/crio-780ba6abc214dc75b5e34ff7ee211aaa825fce3269dcbf3f016f263be9527b46 WatchSource:0}: Error finding container 780ba6abc214dc75b5e34ff7ee211aaa825fce3269dcbf3f016f263be9527b46: Status 404 returned error can't find the container with id 780ba6abc214dc75b5e34ff7ee211aaa825fce3269dcbf3f016f263be9527b46
Sep 09 20:22:47.790118 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:47.790082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z" event={"ID":"03d79687-ff1c-49fb-ae29-37e8901a5e68","Type":"ContainerStarted","Data":"780ba6abc214dc75b5e34ff7ee211aaa825fce3269dcbf3f016f263be9527b46"}
Sep 09 20:22:49.127227 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:49.127186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:49.208844 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:49.208722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:50.293194 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:50.293145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:50.803013 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:50.802979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z" event={"ID":"03d79687-ff1c-49fb-ae29-37e8901a5e68","Type":"ContainerStarted","Data":"3d516399f7e3fd09638bbac444204f8a134da71192eb80366080f578442d3b8a"}
Sep 09 20:22:50.834885 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:50.834830    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-758cfc744b-w7r8z" podStartSLOduration=0.909554762 podStartE2EDuration="3.83481468s" totalImagesPullingTime="2.925259918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:50.833093628 +0000 UTC m=+378.072195929" watchObservedRunningTime="2026-09-09 20:22:50.83481468 +0000 UTC m=+378.073916980"
Sep 09 20:22:54.459696 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:54.459657    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:54.558942 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:54.558908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:55.109326 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.109295    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"]
Sep 09 20:22:55.111616 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.111589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.117992 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.114681    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8jbcl\""
Sep 09 20:22:55.117992 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.114707    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:22:55.131426 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.131400    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"]
Sep 09 20:22:55.146107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.146072    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:55.278052 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.278013    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.278052 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.278052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.278277 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.278075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.278277 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.278100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.278277 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.278170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45323d30-028c-479f-8086-28e773041b10-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.278277 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.278220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb2st\" (UniqueName: \"kubernetes.io/projected/45323d30-028c-479f-8086-28e773041b10-kube-api-access-hb2st\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.278425 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.278277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.278425 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.278299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45323d30-028c-479f-8086-28e773041b10-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.278425 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.278330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45323d30-028c-479f-8086-28e773041b10-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.379500 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.379401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.379500 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.379448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.379500 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.379478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.379806 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.379512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45323d30-028c-479f-8086-28e773041b10-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.379806 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.379546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb2st\" (UniqueName: \"kubernetes.io/projected/45323d30-028c-479f-8086-28e773041b10-kube-api-access-hb2st\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.379806 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.379720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.379806 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.379791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45323d30-028c-479f-8086-28e773041b10-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.380006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.379834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45323d30-028c-479f-8086-28e773041b10-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.380006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.379849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.380006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.379903    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.380006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.379922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.380220 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.380179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.380357 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.380338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.380523 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.380506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45323d30-028c-479f-8086-28e773041b10-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.382310 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.382279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45323d30-028c-479f-8086-28e773041b10-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.382310 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.382308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45323d30-028c-479f-8086-28e773041b10-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.392567 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.392546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45323d30-028c-479f-8086-28e773041b10-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.392700 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.392647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb2st\" (UniqueName: \"kubernetes.io/projected/45323d30-028c-479f-8086-28e773041b10-kube-api-access-hb2st\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5\" (UID: \"45323d30-028c-479f-8086-28e773041b10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.405313 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.405274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:55.425010 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.424985    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:55.507032 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.506993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:55.696534 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.696464    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"]
Sep 09 20:22:55.699517 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:22:55.699493    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45323d30_028c_479f_8086_28e773041b10.slice/crio-b4d20167103131e66caa4454e88664f9b6f5ba05c27b7f59a60889130f1ceea9 WatchSource:0}: Error finding container b4d20167103131e66caa4454e88664f9b6f5ba05c27b7f59a60889130f1ceea9: Status 404 returned error can't find the container with id b4d20167103131e66caa4454e88664f9b6f5ba05c27b7f59a60889130f1ceea9
Sep 09 20:22:55.820302 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:55.820265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5" event={"ID":"45323d30-028c-479f-8086-28e773041b10","Type":"ContainerStarted","Data":"b4d20167103131e66caa4454e88664f9b6f5ba05c27b7f59a60889130f1ceea9"}
Sep 09 20:22:57.718157 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:57.718125    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z8k8d"]
Sep 09 20:22:57.722925 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:57.722898    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d"
Sep 09 20:22:57.725515 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:57.725492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 09 20:22:57.725977 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:57.725952    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2hdrf\""
Sep 09 20:22:57.731834 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:57.731780    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z8k8d"]
Sep 09 20:22:57.751742 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:57.751640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:57.801277 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:57.801243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1fa4292-059b-40a2-a8ca-86364d7f8033-cert\") pod \"odh-model-controller-7998c67dc4-z8k8d\" (UID: \"c1fa4292-059b-40a2-a8ca-86364d7f8033\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d"
Sep 09 20:22:57.801444 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:57.801309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m28t9\" (UniqueName: \"kubernetes.io/projected/c1fa4292-059b-40a2-a8ca-86364d7f8033-kube-api-access-m28t9\") pod \"odh-model-controller-7998c67dc4-z8k8d\" (UID: \"c1fa4292-059b-40a2-a8ca-86364d7f8033\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d"
Sep 09 20:22:57.846374 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:57.846301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:57.902320 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:57.902277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1fa4292-059b-40a2-a8ca-86364d7f8033-cert\") pod \"odh-model-controller-7998c67dc4-z8k8d\" (UID: \"c1fa4292-059b-40a2-a8ca-86364d7f8033\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d"
Sep 09 20:22:57.902511 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:57.902352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m28t9\" (UniqueName: \"kubernetes.io/projected/c1fa4292-059b-40a2-a8ca-86364d7f8033-kube-api-access-m28t9\") pod \"odh-model-controller-7998c67dc4-z8k8d\" (UID: \"c1fa4292-059b-40a2-a8ca-86364d7f8033\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d"
Sep 09 20:22:57.902511 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:22:57.902453    2563 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 20:22:57.902640 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:22:57.902526    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1fa4292-059b-40a2-a8ca-86364d7f8033-cert podName:c1fa4292-059b-40a2-a8ca-86364d7f8033 nodeName:}" failed. No retries permitted until 2026-09-09 20:22:58.402505499 +0000 UTC m=+385.641607784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c1fa4292-059b-40a2-a8ca-86364d7f8033-cert") pod "odh-model-controller-7998c67dc4-z8k8d" (UID: "c1fa4292-059b-40a2-a8ca-86364d7f8033") : secret "odh-model-controller-webhook-cert" not found
Sep 09 20:22:57.917191 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:57.917161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28t9\" (UniqueName: \"kubernetes.io/projected/c1fa4292-059b-40a2-a8ca-86364d7f8033-kube-api-access-m28t9\") pod \"odh-model-controller-7998c67dc4-z8k8d\" (UID: \"c1fa4292-059b-40a2-a8ca-86364d7f8033\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d"
Sep 09 20:22:57.940978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:57.940934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:58.233261 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.233214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:58.350055 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.350019    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:58.389148 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.389105    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:22:58.389267 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.389204    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:22:58.389267 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.389248    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:22:58.406539 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.406511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1fa4292-059b-40a2-a8ca-86364d7f8033-cert\") pod \"odh-model-controller-7998c67dc4-z8k8d\" (UID: \"c1fa4292-059b-40a2-a8ca-86364d7f8033\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d"
Sep 09 20:22:58.409313 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.409291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1fa4292-059b-40a2-a8ca-86364d7f8033-cert\") pod \"odh-model-controller-7998c67dc4-z8k8d\" (UID: \"c1fa4292-059b-40a2-a8ca-86364d7f8033\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d"
Sep 09 20:22:58.440966 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.440938    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:58.555742 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.555654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:58.635841 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.635803    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d"
Sep 09 20:22:58.653039 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.653004    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:58.812112 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.812028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:58.835146 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.835110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5" event={"ID":"45323d30-028c-479f-8086-28e773041b10","Type":"ContainerStarted","Data":"b6967cd7c63e8ef19125c8a8ec97d37a2d23485d96b7da89f8ee265dc1d7c4ca"}
Sep 09 20:22:58.856162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.856116    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5" podStartSLOduration=1.168892824 podStartE2EDuration="3.856099094s" totalImagesPullingTime="2.68720627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:58.855498952 +0000 UTC m=+386.094601257" watchObservedRunningTime="2026-09-09 20:22:58.856099094 +0000 UTC m=+386.095201420"
Sep 09 20:22:58.911507 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:58.911469    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z8k8d"]
Sep 09 20:22:58.914395 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:22:58.914367    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fa4292_059b_40a2_a8ca_86364d7f8033.slice/crio-c856a758d85b1c689c854f50f502400a3513ae284a004fba7b407915ebdd35a5 WatchSource:0}: Error finding container c856a758d85b1c689c854f50f502400a3513ae284a004fba7b407915ebdd35a5: Status 404 returned error can't find the container with id c856a758d85b1c689c854f50f502400a3513ae284a004fba7b407915ebdd35a5
Sep 09 20:22:59.425765 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:59.425725    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:59.430216 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:59.430193    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:59.841656 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:59.841615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d" event={"ID":"c1fa4292-059b-40a2-a8ca-86364d7f8033","Type":"ContainerStarted","Data":"c856a758d85b1c689c854f50f502400a3513ae284a004fba7b407915ebdd35a5"}
Sep 09 20:22:59.842253 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:59.842222    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:22:59.843249 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:22:59.843224    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5"
Sep 09 20:23:01.851080 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:01.851041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d" event={"ID":"c1fa4292-059b-40a2-a8ca-86364d7f8033","Type":"ContainerStarted","Data":"f23a089f7c9e8979dc2bdf57578b24e4a7f1a51b4724b425de1d610251e34670"}
Sep 09 20:23:01.867848 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:01.867763    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d" podStartSLOduration=2.015736955 podStartE2EDuration="4.867747894s" totalImagesPullingTime="2.852010939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:01.867654467 +0000 UTC m=+389.106756767" watchObservedRunningTime="2026-09-09 20:23:01.867747894 +0000 UTC m=+389.106850193"
Sep 09 20:23:02.855291 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:02.855254    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d"
Sep 09 20:23:04.454139 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.454103    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"]
Sep 09 20:23:04.456459 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.456439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.462478 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.462437    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4srjj\""
Sep 09 20:23:04.463238 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.463217    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 20:23:04.463336 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.463275    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 20:23:04.463336 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.463280    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 20:23:04.480652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.480622    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"]
Sep 09 20:23:04.562655 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.562620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbgg5\" (UniqueName: \"kubernetes.io/projected/3401c08b-6b18-4a85-94b7-5b07118d346b-kube-api-access-nbgg5\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.562822 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.562662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3401c08b-6b18-4a85-94b7-5b07118d346b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.562822 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.562689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3401c08b-6b18-4a85-94b7-5b07118d346b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.562822 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.562761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3401c08b-6b18-4a85-94b7-5b07118d346b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.562928 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.562843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3401c08b-6b18-4a85-94b7-5b07118d346b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.562928 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.562871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3401c08b-6b18-4a85-94b7-5b07118d346b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.562928 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.562890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3401c08b-6b18-4a85-94b7-5b07118d346b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.663427 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.663392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3401c08b-6b18-4a85-94b7-5b07118d346b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.663613 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.663455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3401c08b-6b18-4a85-94b7-5b07118d346b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.663613 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.663483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3401c08b-6b18-4a85-94b7-5b07118d346b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.663613 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.663500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3401c08b-6b18-4a85-94b7-5b07118d346b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.663613 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.663528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbgg5\" (UniqueName: \"kubernetes.io/projected/3401c08b-6b18-4a85-94b7-5b07118d346b-kube-api-access-nbgg5\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.663613 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.663554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3401c08b-6b18-4a85-94b7-5b07118d346b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.663866 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.663781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3401c08b-6b18-4a85-94b7-5b07118d346b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.664228 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.664197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3401c08b-6b18-4a85-94b7-5b07118d346b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.665902 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.665878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3401c08b-6b18-4a85-94b7-5b07118d346b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.666030 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.666004    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3401c08b-6b18-4a85-94b7-5b07118d346b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.666207 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.666191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3401c08b-6b18-4a85-94b7-5b07118d346b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.666256 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.666240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3401c08b-6b18-4a85-94b7-5b07118d346b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.672494 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.672470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3401c08b-6b18-4a85-94b7-5b07118d346b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.672494 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.672493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbgg5\" (UniqueName: \"kubernetes.io/projected/3401c08b-6b18-4a85-94b7-5b07118d346b-kube-api-access-nbgg5\") pod \"istiod-openshift-gateway-5677ccb49b-qqz4k\" (UID: \"3401c08b-6b18-4a85-94b7-5b07118d346b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:04.766158 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:04.766119    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:05.078152 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:23:05.078119    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3401c08b_6b18_4a85_94b7_5b07118d346b.slice/crio-33ddfd6ce15a4117e63dc9851e92237a411f0af624cf9413002eb64a21da6a8b WatchSource:0}: Error finding container 33ddfd6ce15a4117e63dc9851e92237a411f0af624cf9413002eb64a21da6a8b: Status 404 returned error can't find the container with id 33ddfd6ce15a4117e63dc9851e92237a411f0af624cf9413002eb64a21da6a8b
Sep 09 20:23:05.085479 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:05.085451    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"]
Sep 09 20:23:05.867101 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:05.867057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k" event={"ID":"3401c08b-6b18-4a85-94b7-5b07118d346b","Type":"ContainerStarted","Data":"33ddfd6ce15a4117e63dc9851e92237a411f0af624cf9413002eb64a21da6a8b"}
Sep 09 20:23:07.432151 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:07.432118    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:07.621275 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:07.621237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:07.771556 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:07.771459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:07.871224 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:07.871169    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:08.009937 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:08.009897    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:08.086337 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:08.086303    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:23:08.086453 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:08.086380    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:23:08.880496 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:08.880453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k" event={"ID":"3401c08b-6b18-4a85-94b7-5b07118d346b","Type":"ContainerStarted","Data":"8c6ed7c0eb7ca8879909e80e93aaad2675f302457e26eccbe0305084417b1277"}
Sep 09 20:23:08.880882 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:08.880601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:08.921721 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:08.921672    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k" podStartSLOduration=1.9162527790000001 podStartE2EDuration="4.921654302s" totalImagesPullingTime="3.005401523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:08.919069732 +0000 UTC m=+396.158172033" watchObservedRunningTime="2026-09-09 20:23:08.921654302 +0000 UTC m=+396.160756602"
Sep 09 20:23:09.074051 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:09.074015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:09.214964 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:09.214929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:09.890417 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:09.890388    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qqz4k"
Sep 09 20:23:13.861784 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:13.861754    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-z8k8d"
Sep 09 20:23:39.739630 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:39.739525    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:52.964517 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:52.964464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:53.065068 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:53.065025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:58.819516 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:58.819409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:58.919902 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:58.919867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:59.906971 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:23:59.906934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:00.007996 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:00.007963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:27.690254 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:27.690209    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:30.084758 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:30.084714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:30.498162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:30.498114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:32.218616 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:32.218565    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zjkjd"]
Sep 09 20:24:32.220767 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:32.220748    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-zjkjd"
Sep 09 20:24:32.223182 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:32.223159    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 20:24:32.223299 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:32.223171    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ltrtf\""
Sep 09 20:24:32.223299 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:32.223206    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 20:24:32.235252 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:32.235221    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zjkjd"]
Sep 09 20:24:32.262945 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:32.262895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5865l\" (UniqueName: \"kubernetes.io/projected/efcef3f7-8aea-4cb6-8b9f-e05ad436ab52-kube-api-access-5865l\") pod \"authorino-operator-76b478bd78-zjkjd\" (UID: \"efcef3f7-8aea-4cb6-8b9f-e05ad436ab52\") " pod="kuadrant-system/authorino-operator-76b478bd78-zjkjd"
Sep 09 20:24:32.363960 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:32.363922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5865l\" (UniqueName: \"kubernetes.io/projected/efcef3f7-8aea-4cb6-8b9f-e05ad436ab52-kube-api-access-5865l\") pod \"authorino-operator-76b478bd78-zjkjd\" (UID: \"efcef3f7-8aea-4cb6-8b9f-e05ad436ab52\") " pod="kuadrant-system/authorino-operator-76b478bd78-zjkjd"
Sep 09 20:24:32.384188 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:32.384147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5865l\" (UniqueName: \"kubernetes.io/projected/efcef3f7-8aea-4cb6-8b9f-e05ad436ab52-kube-api-access-5865l\") pod \"authorino-operator-76b478bd78-zjkjd\" (UID: \"efcef3f7-8aea-4cb6-8b9f-e05ad436ab52\") " pod="kuadrant-system/authorino-operator-76b478bd78-zjkjd"
Sep 09 20:24:32.531471 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:32.531385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-zjkjd"
Sep 09 20:24:32.625548 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:32.625514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:32.812704 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:32.812627    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zjkjd"]
Sep 09 20:24:32.817320 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:24:32.817287    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefcef3f7_8aea_4cb6_8b9f_e05ad436ab52.slice/crio-1fa8b3bc77633ebf0daba08b49ce63c5586439ae03decda3379a4afea8bbf16c WatchSource:0}: Error finding container 1fa8b3bc77633ebf0daba08b49ce63c5586439ae03decda3379a4afea8bbf16c: Status 404 returned error can't find the container with id 1fa8b3bc77633ebf0daba08b49ce63c5586439ae03decda3379a4afea8bbf16c
Sep 09 20:24:33.095231 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:33.095115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:33.182784 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:33.182753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zjkjd" event={"ID":"efcef3f7-8aea-4cb6-8b9f-e05ad436ab52","Type":"ContainerStarted","Data":"1fa8b3bc77633ebf0daba08b49ce63c5586439ae03decda3379a4afea8bbf16c"}
Sep 09 20:24:33.525509 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:33.525467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:33.899420 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:33.899337    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:34.425733 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:34.425694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:35.414824 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:35.414798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:35.522098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:35.522065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:36.197821 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:36.197781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zjkjd" event={"ID":"efcef3f7-8aea-4cb6-8b9f-e05ad436ab52","Type":"ContainerStarted","Data":"80fc20d30843bd39e6d96080a61499352bf323ad71829e79e103342c068d584c"}
Sep 09 20:24:36.197997 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:36.197925    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-zjkjd"
Sep 09 20:24:36.216953 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:36.216906    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-zjkjd" podStartSLOduration=1.672860947 podStartE2EDuration="4.216891322s" totalImagesPullingTime="2.544030375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:36.215087117 +0000 UTC m=+483.454189416" watchObservedRunningTime="2026-09-09 20:24:36.216891322 +0000 UTC m=+483.455993622"
Sep 09 20:24:36.989424 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:36.989380    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"]
Sep 09 20:24:36.991687 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:36.991661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:37.001019 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:37.001001    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qngs9\""
Sep 09 20:24:37.012065 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:37.012044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"]
Sep 09 20:24:37.101280 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:37.101239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dws6t\" (UID: \"b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:37.101459 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:37.101300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpm2\" (UniqueName: \"kubernetes.io/projected/b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241-kube-api-access-4vpm2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dws6t\" (UID: \"b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:37.201857 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:37.201823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dws6t\" (UID: \"b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:37.202021 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:37.201894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpm2\" (UniqueName: \"kubernetes.io/projected/b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241-kube-api-access-4vpm2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dws6t\" (UID: \"b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:37.202180 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:37.202164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dws6t\" (UID: \"b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:37.210763 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:37.210739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpm2\" (UniqueName: \"kubernetes.io/projected/b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241-kube-api-access-4vpm2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dws6t\" (UID: \"b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:37.300821 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:37.300727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:37.546558 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:37.546525    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:37.593922 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:37.593897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"]
Sep 09 20:24:37.596252 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:24:37.596223    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d7d3c7_bf8f_4d5c_a8c5_e38e9470c241.slice/crio-f0efa1e0d7d5670c2cdbb2c2099a6a5f993cfa25a5c5db2174b7ebf506d7745d WatchSource:0}: Error finding container f0efa1e0d7d5670c2cdbb2c2099a6a5f993cfa25a5c5db2174b7ebf506d7745d: Status 404 returned error can't find the container with id f0efa1e0d7d5670c2cdbb2c2099a6a5f993cfa25a5c5db2174b7ebf506d7745d
Sep 09 20:24:37.647224 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:37.647185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:38.207219 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:38.207177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t" event={"ID":"b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241","Type":"ContainerStarted","Data":"f0efa1e0d7d5670c2cdbb2c2099a6a5f993cfa25a5c5db2174b7ebf506d7745d"}
Sep 09 20:24:40.503745 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:40.503696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:40.602215 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:40.602169    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:42.226029 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:42.225993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t" event={"ID":"b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241","Type":"ContainerStarted","Data":"a11741e47aaa41b01c49820a702c60e43b93627e310aa63b586ace4e42b75ed9"}
Sep 09 20:24:42.226406 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:42.226115    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:42.248355 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:42.248289    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t" podStartSLOduration=2.442159039 podStartE2EDuration="6.248271738s" totalImagesPullingTime="3.806112699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:42.245719203 +0000 UTC m=+489.484821503" watchObservedRunningTime="2026-09-09 20:24:42.248271738 +0000 UTC m=+489.487374040"
Sep 09 20:24:43.906257 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:43.906218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:44.147653 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:44.147616    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:44.313529 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:44.313491    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:44.379525 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:44.379486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:44.495986 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:44.495949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:44.599035 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:44.598954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:47.203940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:47.203911    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-zjkjd"
Sep 09 20:24:48.630208 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:48.630175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:49.544069 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:49.544034    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:49.642723 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:49.642690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:49.743033 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:49.742985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:53.232407 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:53.232373    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:55.561480 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.561448    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"]
Sep 09 20:24:55.561909 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.561662    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t" podUID="b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241" containerName="manager" containerID="cri-o://a11741e47aaa41b01c49820a702c60e43b93627e310aa63b586ace4e42b75ed9" gracePeriod=2
Sep 09 20:24:55.576692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.576664    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"]
Sep 09 20:24:55.590547 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:24:55.590506    2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-dws6t\": pods \"kuadrant-operator-controller-manager-76f8f686bf-dws6t\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:55.612816 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.612784    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"]
Sep 09 20:24:55.613156 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.613128    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241" containerName="manager"
Sep 09 20:24:55.613156 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.613147    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241" containerName="manager"
Sep 09 20:24:55.613307 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.613196    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241" containerName="manager"
Sep 09 20:24:55.614873 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.614856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"
Sep 09 20:24:55.665127 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.665091    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"]
Sep 09 20:24:55.691645 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:24:55.691604    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dws6t\" is forbidden: User \"system:node:ip-10-0-143-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-241.ec2.internal' and this object" podUID="b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:55.751603 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.751549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3cb3e8c-a3d0-476c-9c0c-4742392e1071-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-s2dqk\" (UID: \"a3cb3e8c-a3d0-476c-9c0c-4742392e1071\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"
Sep 09 20:24:55.751777 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.751681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhxcv\" (UniqueName: \"kubernetes.io/projected/a3cb3e8c-a3d0-476c-9c0c-4742392e1071-kube-api-access-nhxcv\") pod \"kuadrant-operator-controller-manager-5f685c496f-s2dqk\" (UID: \"a3cb3e8c-a3d0-476c-9c0c-4742392e1071\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"
Sep 09 20:24:55.806440 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.806418    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:55.808625 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:24:55.808601    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dws6t\" is forbidden: User \"system:node:ip-10-0-143-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-241.ec2.internal' and this object" podUID="b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:55.852878 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.852820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3cb3e8c-a3d0-476c-9c0c-4742392e1071-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-s2dqk\" (UID: \"a3cb3e8c-a3d0-476c-9c0c-4742392e1071\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"
Sep 09 20:24:55.852878 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.852870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhxcv\" (UniqueName: \"kubernetes.io/projected/a3cb3e8c-a3d0-476c-9c0c-4742392e1071-kube-api-access-nhxcv\") pod \"kuadrant-operator-controller-manager-5f685c496f-s2dqk\" (UID: \"a3cb3e8c-a3d0-476c-9c0c-4742392e1071\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"
Sep 09 20:24:55.853163 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.853144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3cb3e8c-a3d0-476c-9c0c-4742392e1071-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-s2dqk\" (UID: \"a3cb3e8c-a3d0-476c-9c0c-4742392e1071\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"
Sep 09 20:24:55.862918 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.862888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhxcv\" (UniqueName: \"kubernetes.io/projected/a3cb3e8c-a3d0-476c-9c0c-4742392e1071-kube-api-access-nhxcv\") pod \"kuadrant-operator-controller-manager-5f685c496f-s2dqk\" (UID: \"a3cb3e8c-a3d0-476c-9c0c-4742392e1071\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"
Sep 09 20:24:55.941891 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.941844    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"
Sep 09 20:24:55.953985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.953963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vpm2\" (UniqueName: \"kubernetes.io/projected/b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241-kube-api-access-4vpm2\") pod \"b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241\" (UID: \"b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241\") "
Sep 09 20:24:55.954073 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.954042    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241-extensions-socket-volume\") pod \"b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241\" (UID: \"b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241\") "
Sep 09 20:24:55.954479 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.954453    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241" (UID: "b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:55.956194 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:55.956157    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241-kube-api-access-4vpm2" (OuterVolumeSpecName: "kube-api-access-4vpm2") pod "b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241" (UID: "b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241"). InnerVolumeSpecName "kube-api-access-4vpm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:56.055013 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.054981    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241-extensions-socket-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:24:56.055013 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.055009    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vpm2\" (UniqueName: \"kubernetes.io/projected/b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241-kube-api-access-4vpm2\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:24:56.087076 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.087039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:56.188303 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.188210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:56.222236 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.222208    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"]
Sep 09 20:24:56.225319 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:24:56.225293    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3cb3e8c_a3d0_476c_9c0c_4742392e1071.slice/crio-bcbacccc27638f46d175ab9d3023e107ffa92cf7a1ce03c1f669d701e13014e0 WatchSource:0}: Error finding container bcbacccc27638f46d175ab9d3023e107ffa92cf7a1ce03c1f669d701e13014e0: Status 404 returned error can't find the container with id bcbacccc27638f46d175ab9d3023e107ffa92cf7a1ce03c1f669d701e13014e0
Sep 09 20:24:56.287955 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.287923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk" event={"ID":"a3cb3e8c-a3d0-476c-9c0c-4742392e1071","Type":"ContainerStarted","Data":"c13735a2368926bdac569d52ca23877b49aae4a3a1587ce07d0755bc99b67131"}
Sep 09 20:24:56.288098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.288004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk" event={"ID":"a3cb3e8c-a3d0-476c-9c0c-4742392e1071","Type":"ContainerStarted","Data":"bcbacccc27638f46d175ab9d3023e107ffa92cf7a1ce03c1f669d701e13014e0"}
Sep 09 20:24:56.288098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.288035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"
Sep 09 20:24:56.289217 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.289193    2563 generic.go:356] "Generic (PLEG): container finished" podID="b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241" containerID="a11741e47aaa41b01c49820a702c60e43b93627e310aa63b586ace4e42b75ed9" exitCode=0
Sep 09 20:24:56.289335 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.289233    2563 scope.go:117] "RemoveContainer" containerID="a11741e47aaa41b01c49820a702c60e43b93627e310aa63b586ace4e42b75ed9"
Sep 09 20:24:56.289335 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.289245    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:56.297747 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.297725    2563 scope.go:117] "RemoveContainer" containerID="a11741e47aaa41b01c49820a702c60e43b93627e310aa63b586ace4e42b75ed9"
Sep 09 20:24:56.298007 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:24:56.297987    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11741e47aaa41b01c49820a702c60e43b93627e310aa63b586ace4e42b75ed9\": container with ID starting with a11741e47aaa41b01c49820a702c60e43b93627e310aa63b586ace4e42b75ed9 not found: ID does not exist" containerID="a11741e47aaa41b01c49820a702c60e43b93627e310aa63b586ace4e42b75ed9"
Sep 09 20:24:56.298051 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.298015    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11741e47aaa41b01c49820a702c60e43b93627e310aa63b586ace4e42b75ed9"} err="failed to get container status \"a11741e47aaa41b01c49820a702c60e43b93627e310aa63b586ace4e42b75ed9\": rpc error: code = NotFound desc = could not find container \"a11741e47aaa41b01c49820a702c60e43b93627e310aa63b586ace4e42b75ed9\": container with ID starting with a11741e47aaa41b01c49820a702c60e43b93627e310aa63b586ace4e42b75ed9 not found: ID does not exist"
Sep 09 20:24:56.309936 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:56.309899    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk" podStartSLOduration=1.309885798 podStartE2EDuration="1.309885798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:56.308558366 +0000 UTC m=+503.547660666" watchObservedRunningTime="2026-09-09 20:24:56.309885798 +0000 UTC m=+503.548988097"
Sep 09 20:24:56.310582 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:24:56.310549    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dws6t\" is forbidden: User \"system:node:ip-10-0-143-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-241.ec2.internal' and this object" podUID="b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:56.312610 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:24:56.312568    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dws6t\" is forbidden: User \"system:node:ip-10-0-143-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-241.ec2.internal' and this object" podUID="b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dws6t"
Sep 09 20:24:57.313060 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:24:57.313019    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241" path="/var/lib/kubelet/pods/b9d7d3c7-bf8f-4d5c-a8c5-e38e9470c241/volumes"
Sep 09 20:25:00.408230 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:00.408201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:00.503933 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:00.503884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:04.208951 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:04.208898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:04.411704 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:04.411655    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:04.477256 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:04.477157    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:07.297136 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:07.297105    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"
Sep 09 20:25:07.738828 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:07.738784    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"]
Sep 09 20:25:07.745092 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:07.745067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"
Sep 09 20:25:07.768747 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:07.768714    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"]
Sep 09 20:25:07.853183 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:07.853149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8ef4400-3124-45b5-8a46-402b7b38f6e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lxmtj\" (UID: \"d8ef4400-3124-45b5-8a46-402b7b38f6e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"
Sep 09 20:25:07.853367 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:07.853210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsr7q\" (UniqueName: \"kubernetes.io/projected/d8ef4400-3124-45b5-8a46-402b7b38f6e4-kube-api-access-vsr7q\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lxmtj\" (UID: \"d8ef4400-3124-45b5-8a46-402b7b38f6e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"
Sep 09 20:25:07.953858 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:07.953772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsr7q\" (UniqueName: \"kubernetes.io/projected/d8ef4400-3124-45b5-8a46-402b7b38f6e4-kube-api-access-vsr7q\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lxmtj\" (UID: \"d8ef4400-3124-45b5-8a46-402b7b38f6e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"
Sep 09 20:25:07.953858 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:07.953830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8ef4400-3124-45b5-8a46-402b7b38f6e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lxmtj\" (UID: \"d8ef4400-3124-45b5-8a46-402b7b38f6e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"
Sep 09 20:25:07.954147 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:07.954132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8ef4400-3124-45b5-8a46-402b7b38f6e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lxmtj\" (UID: \"d8ef4400-3124-45b5-8a46-402b7b38f6e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"
Sep 09 20:25:07.967241 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:07.967199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsr7q\" (UniqueName: \"kubernetes.io/projected/d8ef4400-3124-45b5-8a46-402b7b38f6e4-kube-api-access-vsr7q\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lxmtj\" (UID: \"d8ef4400-3124-45b5-8a46-402b7b38f6e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"
Sep 09 20:25:08.054948 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:08.054908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"
Sep 09 20:25:08.333252 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:08.333224    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"]
Sep 09 20:25:08.335988 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:25:08.335958    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ef4400_3124_45b5_8a46_402b7b38f6e4.slice/crio-24f38ec259a7220ec57436aba5ab845b416523b234c0ff1401f56d7a2686d513 WatchSource:0}: Error finding container 24f38ec259a7220ec57436aba5ab845b416523b234c0ff1401f56d7a2686d513: Status 404 returned error can't find the container with id 24f38ec259a7220ec57436aba5ab845b416523b234c0ff1401f56d7a2686d513
Sep 09 20:25:09.343889 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:09.343836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj" event={"ID":"d8ef4400-3124-45b5-8a46-402b7b38f6e4","Type":"ContainerStarted","Data":"fdfea039764fd0c2ea55cc6f275268877e85bddf92e03bb606bc2a21bb85d851"}
Sep 09 20:25:09.344386 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:09.343906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj" event={"ID":"d8ef4400-3124-45b5-8a46-402b7b38f6e4","Type":"ContainerStarted","Data":"24f38ec259a7220ec57436aba5ab845b416523b234c0ff1401f56d7a2686d513"}
Sep 09 20:25:09.344386 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:09.343936    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"
Sep 09 20:25:09.370447 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:09.370392    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj" podStartSLOduration=2.370375373 podStartE2EDuration="2.370375373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:09.369041037 +0000 UTC m=+516.608143338" watchObservedRunningTime="2026-09-09 20:25:09.370375373 +0000 UTC m=+516.609477672"
Sep 09 20:25:09.811015 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:09.810978    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:09.912219 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:09.912182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:10.012365 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:10.012329    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:10.115959 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:10.115874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:10.225218 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:10.225176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:10.325189 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:10.325157    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:16.187329 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:16.187284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:16.284142 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:16.284106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:20.349996 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:20.349965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"
Sep 09 20:25:20.429471 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:20.429434    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"]
Sep 09 20:25:20.429739 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:20.429715    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk" podUID="a3cb3e8c-a3d0-476c-9c0c-4742392e1071" containerName="manager" containerID="cri-o://c13735a2368926bdac569d52ca23877b49aae4a3a1587ce07d0755bc99b67131" gracePeriod=10
Sep 09 20:25:20.680921 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:20.680889    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"
Sep 09 20:25:20.862005 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:20.861960    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhxcv\" (UniqueName: \"kubernetes.io/projected/a3cb3e8c-a3d0-476c-9c0c-4742392e1071-kube-api-access-nhxcv\") pod \"a3cb3e8c-a3d0-476c-9c0c-4742392e1071\" (UID: \"a3cb3e8c-a3d0-476c-9c0c-4742392e1071\") "
Sep 09 20:25:20.862167 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:20.862021    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3cb3e8c-a3d0-476c-9c0c-4742392e1071-extensions-socket-volume\") pod \"a3cb3e8c-a3d0-476c-9c0c-4742392e1071\" (UID: \"a3cb3e8c-a3d0-476c-9c0c-4742392e1071\") "
Sep 09 20:25:20.862441 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:20.862405    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cb3e8c-a3d0-476c-9c0c-4742392e1071-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a3cb3e8c-a3d0-476c-9c0c-4742392e1071" (UID: "a3cb3e8c-a3d0-476c-9c0c-4742392e1071"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:20.864201 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:20.864168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3cb3e8c-a3d0-476c-9c0c-4742392e1071-kube-api-access-nhxcv" (OuterVolumeSpecName: "kube-api-access-nhxcv") pod "a3cb3e8c-a3d0-476c-9c0c-4742392e1071" (UID: "a3cb3e8c-a3d0-476c-9c0c-4742392e1071"). InnerVolumeSpecName "kube-api-access-nhxcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:25:20.962968 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:20.962938    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhxcv\" (UniqueName: \"kubernetes.io/projected/a3cb3e8c-a3d0-476c-9c0c-4742392e1071-kube-api-access-nhxcv\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:25:20.962968 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:20.962967    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3cb3e8c-a3d0-476c-9c0c-4742392e1071-extensions-socket-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:25:21.391740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:21.391704    2563 generic.go:356] "Generic (PLEG): container finished" podID="a3cb3e8c-a3d0-476c-9c0c-4742392e1071" containerID="c13735a2368926bdac569d52ca23877b49aae4a3a1587ce07d0755bc99b67131" exitCode=0
Sep 09 20:25:21.392189 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:21.391744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk" event={"ID":"a3cb3e8c-a3d0-476c-9c0c-4742392e1071","Type":"ContainerDied","Data":"c13735a2368926bdac569d52ca23877b49aae4a3a1587ce07d0755bc99b67131"}
Sep 09 20:25:21.392189 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:21.391758    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"
Sep 09 20:25:21.392189 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:21.391788    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk" event={"ID":"a3cb3e8c-a3d0-476c-9c0c-4742392e1071","Type":"ContainerDied","Data":"bcbacccc27638f46d175ab9d3023e107ffa92cf7a1ce03c1f669d701e13014e0"}
Sep 09 20:25:21.392189 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:21.391810    2563 scope.go:117] "RemoveContainer" containerID="c13735a2368926bdac569d52ca23877b49aae4a3a1587ce07d0755bc99b67131"
Sep 09 20:25:21.402957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:21.402937    2563 scope.go:117] "RemoveContainer" containerID="c13735a2368926bdac569d52ca23877b49aae4a3a1587ce07d0755bc99b67131"
Sep 09 20:25:21.403305 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:25:21.403284    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13735a2368926bdac569d52ca23877b49aae4a3a1587ce07d0755bc99b67131\": container with ID starting with c13735a2368926bdac569d52ca23877b49aae4a3a1587ce07d0755bc99b67131 not found: ID does not exist" containerID="c13735a2368926bdac569d52ca23877b49aae4a3a1587ce07d0755bc99b67131"
Sep 09 20:25:21.403356 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:21.403314    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13735a2368926bdac569d52ca23877b49aae4a3a1587ce07d0755bc99b67131"} err="failed to get container status \"c13735a2368926bdac569d52ca23877b49aae4a3a1587ce07d0755bc99b67131\": rpc error: code = NotFound desc = could not find container \"c13735a2368926bdac569d52ca23877b49aae4a3a1587ce07d0755bc99b67131\": container with ID starting with c13735a2368926bdac569d52ca23877b49aae4a3a1587ce07d0755bc99b67131 not found: ID does not exist"
Sep 09 20:25:21.411189 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:21.411166    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"]
Sep 09 20:25:21.419775 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:21.419752    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-s2dqk"]
Sep 09 20:25:23.312413 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:23.312378    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3cb3e8c-a3d0-476c-9c0c-4742392e1071" path="/var/lib/kubelet/pods/a3cb3e8c-a3d0-476c-9c0c-4742392e1071/volumes"
Sep 09 20:25:23.900442 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:23.900402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:24.226776 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:24.226740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:38.031907 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.031861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:38.132255 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.132204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:38.214998 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.214958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:38.456097 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.456064    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:25:38.456419 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.456401    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3cb3e8c-a3d0-476c-9c0c-4742392e1071" containerName="manager"
Sep 09 20:25:38.456498 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.456424    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3cb3e8c-a3d0-476c-9c0c-4742392e1071" containerName="manager"
Sep 09 20:25:38.456560 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.456509    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3cb3e8c-a3d0-476c-9c0c-4742392e1071" containerName="manager"
Sep 09 20:25:38.459194 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.459174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"
Sep 09 20:25:38.461724 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.461705    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 20:25:38.461921 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.461902    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xl2tw\""
Sep 09 20:25:38.469827 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.469805    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:25:38.498869 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.498839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2dbc1335-4b11-4ff0-a531-35d43b20dbb3-config-file\") pod \"limitador-limitador-84994f7ffc-8gkmm\" (UID: \"2dbc1335-4b11-4ff0-a531-35d43b20dbb3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"
Sep 09 20:25:38.498869 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.498872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwck\" (UniqueName: \"kubernetes.io/projected/2dbc1335-4b11-4ff0-a531-35d43b20dbb3-kube-api-access-qrwck\") pod \"limitador-limitador-84994f7ffc-8gkmm\" (UID: \"2dbc1335-4b11-4ff0-a531-35d43b20dbb3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"
Sep 09 20:25:38.508731 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.508700    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:25:38.600372 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.600331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2dbc1335-4b11-4ff0-a531-35d43b20dbb3-config-file\") pod \"limitador-limitador-84994f7ffc-8gkmm\" (UID: \"2dbc1335-4b11-4ff0-a531-35d43b20dbb3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"
Sep 09 20:25:38.600372 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.600377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwck\" (UniqueName: \"kubernetes.io/projected/2dbc1335-4b11-4ff0-a531-35d43b20dbb3-kube-api-access-qrwck\") pod \"limitador-limitador-84994f7ffc-8gkmm\" (UID: \"2dbc1335-4b11-4ff0-a531-35d43b20dbb3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"
Sep 09 20:25:38.601135 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.601111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2dbc1335-4b11-4ff0-a531-35d43b20dbb3-config-file\") pod \"limitador-limitador-84994f7ffc-8gkmm\" (UID: \"2dbc1335-4b11-4ff0-a531-35d43b20dbb3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"
Sep 09 20:25:38.609253 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.609221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwck\" (UniqueName: \"kubernetes.io/projected/2dbc1335-4b11-4ff0-a531-35d43b20dbb3-kube-api-access-qrwck\") pod \"limitador-limitador-84994f7ffc-8gkmm\" (UID: \"2dbc1335-4b11-4ff0-a531-35d43b20dbb3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"
Sep 09 20:25:38.770169 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.770081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"
Sep 09 20:25:38.920209 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:38.920172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:39.016731 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:39.016691    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:39.037535 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:39.037512    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:25:39.039899 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:25:39.039856    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbc1335_4b11_4ff0_a531_35d43b20dbb3.slice/crio-c5d13ba63dfa27f5d7b2d0b87f35133f1c98013797b14db0a120fa269a7d4be8 WatchSource:0}: Error finding container c5d13ba63dfa27f5d7b2d0b87f35133f1c98013797b14db0a120fa269a7d4be8: Status 404 returned error can't find the container with id c5d13ba63dfa27f5d7b2d0b87f35133f1c98013797b14db0a120fa269a7d4be8
Sep 09 20:25:39.471534 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:39.471494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8gkmm" event={"ID":"2dbc1335-4b11-4ff0-a531-35d43b20dbb3","Type":"ContainerStarted","Data":"c5d13ba63dfa27f5d7b2d0b87f35133f1c98013797b14db0a120fa269a7d4be8"}
Sep 09 20:25:40.476634 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:40.476594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8gkmm" event={"ID":"2dbc1335-4b11-4ff0-a531-35d43b20dbb3","Type":"ContainerStarted","Data":"ef1476a706461d5f8dedfb22fd5885bdebb017afbe485832f311b6f65f7ff597"}
Sep 09 20:25:40.477117 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:40.476730    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"
Sep 09 20:25:41.656668 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:41.656632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:41.727677 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:41.727644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:48.198127 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:48.198084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:51.482428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:51.482396    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"
Sep 09 20:25:51.499335 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:51.499283    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-8gkmm" podStartSLOduration=12.6889072 podStartE2EDuration="13.499267458s" totalImagesPullingTime="810.360258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:40.511795655 +0000 UTC m=+547.750897954" watchObservedRunningTime="2026-09-09 20:25:51.499267458 +0000 UTC m=+558.738369762"
Sep 09 20:25:55.406546 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:55.406506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:55.506632 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:55.506591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:55.944110 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:25:55.944074    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:08.445906 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:08.445862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:09.164726 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:09.164681    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:15.910850 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:15.910795    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:16.014914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:16.014874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:16.109974 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:16.109936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:16.209433 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:16.209396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:16.310075 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:16.310038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:19.562783 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:19.562744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:19.659796 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:19.659764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:50.293909 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:50.293872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:51.641349 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:51.641295    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:51.734132 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:51.734094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:52.133331 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:52.133296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:52.228258 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:52.228219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:52.327654 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:52.327604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:52.417345 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:26:52.417252    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:27:02.404593 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:27:02.404480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:02.500633 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:27:02.500564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:16.519091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:27:16.519049    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:16.576958 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:27:16.576923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:23.087174 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:27:23.087132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:23.502926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:27:23.502895    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:27:27.582832 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:27:27.582785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:28.207559 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:27:28.207523    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:27:33.784029 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:27:33.783990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:34.605902 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:27:34.605864    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:27:57.195299 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:27:57.195253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:58.399046 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:27:58.399012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:27:59.781356 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:27:59.781316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:10.248180 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.248134    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"]
Sep 09 20:28:10.251777 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.251753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.254466 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.254427    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 20:28:10.254466 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.254427    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xlwgd\""
Sep 09 20:28:10.255229 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.255208    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 20:28:10.255334 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.255250    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 20:28:10.261613 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.261568    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"]
Sep 09 20:28:10.283364 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.283329    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:10.358841 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.358808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.358841 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.358847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.359076 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.358878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.359076 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.358948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jsnk\" (UniqueName: \"kubernetes.io/projected/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-kube-api-access-2jsnk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.359076 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.358991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.359076 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.359019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.460304 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.460262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.460304 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.460302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.460567 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.460325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.460567 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.460360    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jsnk\" (UniqueName: \"kubernetes.io/projected/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-kube-api-access-2jsnk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.460567 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.460384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.460567 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.460408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.460861 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.460801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.460861 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.460829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.460971 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.460947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.462647 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.462628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.463112 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.463093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.469678 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.469651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jsnk\" (UniqueName: \"kubernetes.io/projected/e909a0fa-cce3-4f44-aca4-08bad6e0afe9-kube-api-access-2jsnk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk\" (UID: \"e909a0fa-cce3-4f44-aca4-08bad6e0afe9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.563645 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.563531    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:10.837055 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.837025    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"]
Sep 09 20:28:10.839630 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:28:10.839593    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode909a0fa_cce3_4f44_aca4_08bad6e0afe9.slice/crio-552fe2b080bcf476a490d7eb23ad582633f9dfabf60b62f213e83b7cc5571bc4 WatchSource:0}: Error finding container 552fe2b080bcf476a490d7eb23ad582633f9dfabf60b62f213e83b7cc5571bc4: Status 404 returned error can't find the container with id 552fe2b080bcf476a490d7eb23ad582633f9dfabf60b62f213e83b7cc5571bc4
Sep 09 20:28:10.840870 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:10.840852    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:28:11.089794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:11.089708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk" event={"ID":"e909a0fa-cce3-4f44-aca4-08bad6e0afe9","Type":"ContainerStarted","Data":"552fe2b080bcf476a490d7eb23ad582633f9dfabf60b62f213e83b7cc5571bc4"}
Sep 09 20:28:12.016889 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:12.016849    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:28:12.094844 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:12.094758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk" event={"ID":"e909a0fa-cce3-4f44-aca4-08bad6e0afe9","Type":"ContainerStarted","Data":"5da4641599101a5aee07870462e9382c5fe1020082d86db4fac67a83d64203a7"}
Sep 09 20:28:12.120744 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:12.120690    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk" podStartSLOduration=1.154554532 podStartE2EDuration="2.120672625s" totalImagesPullingTime="966.118093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:28:12.120093421 +0000 UTC m=+699.359195744" watchObservedRunningTime="2026-09-09 20:28:12.120672625 +0000 UTC m=+699.359774926"
Sep 09 20:28:16.946845 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:16.946809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:18.608068 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:18.608018    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:28:20.376472 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:20.376436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:20.564360 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:20.564321    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:20.564559 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:20.564391    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:20.577127 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:20.577099    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:21.145218 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:21.145187    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk"
Sep 09 20:28:28.403880 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:28.403778    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:30.404784 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:30.404749    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:28:47.221280 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:28:47.221242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:19.192757 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:19.192714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:26.938236 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:26.938185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:26.987320 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:26.987287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:34.429270 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:34.429236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:34.527772 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:34.527737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:35.627398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:35.627362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:35.727661 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:35.727602    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:55.090277 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.090191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:55.246603 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.246543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:55.369827 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.369752    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"]
Sep 09 20:29:55.373404 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.373378    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.375705 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.375681    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-21117b-openshift-default-dockercfg-ft89v\""
Sep 09 20:29:55.382924 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.382895    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"]
Sep 09 20:29:55.397013 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.396988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:55.424732 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.424703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-data\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.424869 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.424746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-token\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.424869 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.424776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-credential-socket\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.424869 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.424813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-workload-certs\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.425006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.424891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-workload-socket\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.425006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.424925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-envoy\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.425006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.424951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1314a2b3-c135-4998-99b5-76a9b81370c2-istiod-ca-cert\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.425100 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.425034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqdmq\" (UniqueName: \"kubernetes.io/projected/1314a2b3-c135-4998-99b5-76a9b81370c2-kube-api-access-jqdmq\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.425100 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.425062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-podinfo\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.526088 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.526051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-data\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.526262 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.526096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-token\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.526262 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.526128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-credential-socket\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.526262 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.526162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-workload-certs\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.526262 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.526203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-workload-socket\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.526262 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.526227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-envoy\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.526526 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.526353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1314a2b3-c135-4998-99b5-76a9b81370c2-istiod-ca-cert\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.526526 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.526441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqdmq\" (UniqueName: \"kubernetes.io/projected/1314a2b3-c135-4998-99b5-76a9b81370c2-kube-api-access-jqdmq\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.526526 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.526472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-podinfo\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.526526 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.526496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-data\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.526749 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.526549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-credential-socket\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.526885 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.526862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-workload-certs\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.526956 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.526919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-workload-socket\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.527074 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.527053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1314a2b3-c135-4998-99b5-76a9b81370c2-istiod-ca-cert\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.528620 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.528603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-envoy\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.529188 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.529165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-podinfo\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.533485 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.533455    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-token\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.533759 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.533740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqdmq\" (UniqueName: \"kubernetes.io/projected/1314a2b3-c135-4998-99b5-76a9b81370c2-kube-api-access-jqdmq\") pod \"e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") " pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.688535 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.688446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:55.762458 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.762407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:55.971961 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.971928    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"]
Sep 09 20:29:55.974936 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:29:55.974913    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1314a2b3_c135_4998_99b5_76a9b81370c2.slice/crio-27b9943c3f8c177a039f06e8dc62764c84411a31f1d8e25ff46271aa7f316232 WatchSource:0}: Error finding container 27b9943c3f8c177a039f06e8dc62764c84411a31f1d8e25ff46271aa7f316232: Status 404 returned error can't find the container with id 27b9943c3f8c177a039f06e8dc62764c84411a31f1d8e25ff46271aa7f316232
Sep 09 20:29:55.977618 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.977563    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:29:55.977704 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.977652    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:29:55.977704 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:55.977680    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:29:56.529057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:56.529017    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" event={"ID":"1314a2b3-c135-4998-99b5-76a9b81370c2","Type":"ContainerStarted","Data":"b798fa556afa4f3de4628b0a36fa674b416b896bedb60cb7fd71569bf37fffce"}
Sep 09 20:29:56.529423 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:56.529088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" event={"ID":"1314a2b3-c135-4998-99b5-76a9b81370c2","Type":"ContainerStarted","Data":"27b9943c3f8c177a039f06e8dc62764c84411a31f1d8e25ff46271aa7f316232"}
Sep 09 20:29:56.549609 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:56.549542    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" podStartSLOduration=1.549524135 podStartE2EDuration="1.549524135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:29:56.548234102 +0000 UTC m=+803.787336403" watchObservedRunningTime="2026-09-09 20:29:56.549524135 +0000 UTC m=+803.788626440"
Sep 09 20:29:56.638863 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:56.638826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:56.689093 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:56.689045    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:56.734510 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:56.734473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:57.108797 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:57.108758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:57.694491 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:57.694460    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:58.538520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:58.538491    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:58.539321 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:58.539300    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:29:58.585375 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:58.585334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:58.661995 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:58.661963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:58.912597 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:58.912496    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:29:59.762343 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:59.762304    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:59.855938 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:59.855901    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:59.954907 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:29:59.954871    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:00.132517 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:00.132430    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816430-f8b4k"]
Sep 09 20:30:00.135804 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:00.135781    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816430-f8b4k"
Sep 09 20:30:00.138021 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:00.138003    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:30:00.143686 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:00.143436    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816430-f8b4k"]
Sep 09 20:30:00.147486 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:00.147463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816430-f8b4k"
Sep 09 20:30:00.421322 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:00.421299    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816430-f8b4k"]
Sep 09 20:30:00.423821 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:30:00.423790    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e24d8b_8b52_4b62_8419_a82b3b7cdf16.slice/crio-561bf095d7e18215d90b096180bab6e66de4ad82b3c045c1766dcd2a9d1be2b2 WatchSource:0}: Error finding container 561bf095d7e18215d90b096180bab6e66de4ad82b3c045c1766dcd2a9d1be2b2: Status 404 returned error can't find the container with id 561bf095d7e18215d90b096180bab6e66de4ad82b3c045c1766dcd2a9d1be2b2
Sep 09 20:30:00.485842 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:00.485800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:00.548746 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:00.548711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:00.549318 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:00.549289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816430-f8b4k" event={"ID":"b0e24d8b-8b52-4b62-8419-a82b3b7cdf16","Type":"ContainerStarted","Data":"561bf095d7e18215d90b096180bab6e66de4ad82b3c045c1766dcd2a9d1be2b2"}
Sep 09 20:30:01.734352 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:01.734315    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7589-e2e17e"]
Sep 09 20:30:01.738877 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:01.738853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7589-e2e17e"
Sep 09 20:30:01.741356 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:01.741307    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5fctx\""
Sep 09 20:30:01.741993 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:01.741910    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7589-e2e17e"]
Sep 09 20:30:01.782826 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:01.782784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78szs\" (UniqueName: \"kubernetes.io/projected/1b9f9587-5be9-4ba4-b90e-54d55bc354a1-kube-api-access-78szs\") pod \"test-curl-7589-e2e17e\" (UID: \"1b9f9587-5be9-4ba4-b90e-54d55bc354a1\") " pod="opendatahub/test-curl-7589-e2e17e"
Sep 09 20:30:01.883443 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:01.883405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78szs\" (UniqueName: \"kubernetes.io/projected/1b9f9587-5be9-4ba4-b90e-54d55bc354a1-kube-api-access-78szs\") pod \"test-curl-7589-e2e17e\" (UID: \"1b9f9587-5be9-4ba4-b90e-54d55bc354a1\") " pod="opendatahub/test-curl-7589-e2e17e"
Sep 09 20:30:01.890941 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:01.890905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78szs\" (UniqueName: \"kubernetes.io/projected/1b9f9587-5be9-4ba4-b90e-54d55bc354a1-kube-api-access-78szs\") pod \"test-curl-7589-e2e17e\" (UID: \"1b9f9587-5be9-4ba4-b90e-54d55bc354a1\") " pod="opendatahub/test-curl-7589-e2e17e"
Sep 09 20:30:02.050091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:02.049994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7589-e2e17e"
Sep 09 20:30:02.323195 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:02.323097    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7589-e2e17e"]
Sep 09 20:30:02.325886 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:30:02.325859    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b9f9587_5be9_4ba4_b90e_54d55bc354a1.slice/crio-3451360392c3e6052210ab4533b498481dfc26402f03c43fabff0648cfbd8225 WatchSource:0}: Error finding container 3451360392c3e6052210ab4533b498481dfc26402f03c43fabff0648cfbd8225: Status 404 returned error can't find the container with id 3451360392c3e6052210ab4533b498481dfc26402f03c43fabff0648cfbd8225
Sep 09 20:30:02.559955 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:02.559912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7589-e2e17e" event={"ID":"1b9f9587-5be9-4ba4-b90e-54d55bc354a1","Type":"ContainerStarted","Data":"3451360392c3e6052210ab4533b498481dfc26402f03c43fabff0648cfbd8225"}
Sep 09 20:30:02.831537 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:02.831496    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:02.937141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:02.937101    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:03.074845 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:03.074807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:03.351356 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:03.351316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:04.069280 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:04.069248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:04.570895 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:04.570854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816430-f8b4k" event={"ID":"b0e24d8b-8b52-4b62-8419-a82b3b7cdf16","Type":"ContainerStarted","Data":"10a3b02d88a666dabc291ddddc6dd9df55e804b53c87336756643ad99d73ef5b"}
Sep 09 20:30:04.572379 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:04.572350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7589-e2e17e" event={"ID":"1b9f9587-5be9-4ba4-b90e-54d55bc354a1","Type":"ContainerStarted","Data":"601f4d3358fd44134e94774f68138c286ec4dcf90a2d14cc773c80a49d8d6b8b"}
Sep 09 20:30:04.586692 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:04.586635    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816430-f8b4k" podStartSLOduration=1.028571319 podStartE2EDuration="4.586618673s" totalImagesPullingTime="3.558047354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:04.585104454 +0000 UTC m=+811.824206755" watchObservedRunningTime="2026-09-09 20:30:04.586618673 +0000 UTC m=+811.825720972"
Sep 09 20:30:04.602442 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:04.602386    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7589-e2e17e" podStartSLOduration=1.9490383759999999 podStartE2EDuration="3.602365948s" totalImagesPullingTime="1.653327572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:04.600804504 +0000 UTC m=+811.839906804" watchObservedRunningTime="2026-09-09 20:30:04.602365948 +0000 UTC m=+811.841468248"
Sep 09 20:30:05.179795 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:05.179715    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7589-e2e17e"]
Sep 09 20:30:05.183617 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:05.183561    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7589-e2e17e"]
Sep 09 20:30:05.446388 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:05.446265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:05.577985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:05.577947    2563 generic.go:356] "Generic (PLEG): container finished" podID="b0e24d8b-8b52-4b62-8419-a82b3b7cdf16" containerID="10a3b02d88a666dabc291ddddc6dd9df55e804b53c87336756643ad99d73ef5b" exitCode=0
Sep 09 20:30:05.578157 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:05.578032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816430-f8b4k" event={"ID":"b0e24d8b-8b52-4b62-8419-a82b3b7cdf16","Type":"ContainerDied","Data":"10a3b02d88a666dabc291ddddc6dd9df55e804b53c87336756643ad99d73ef5b"}
Sep 09 20:30:05.578316 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:30:05.578301    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7589-e2e17e" container="test-curl-7589-e2e17e"
Sep 09 20:30:05.713849 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:05.713808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:05.840039 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:05.839999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:05.934479 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:05.934431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:06.008863 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.008763    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"]
Sep 09 20:30:06.009226 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.009211    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b9f9587-5be9-4ba4-b90e-54d55bc354a1" containerName="test-curl-7589-e2e17e"
Sep 09 20:30:06.009273 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.009229    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b9f9587-5be9-4ba4-b90e-54d55bc354a1" containerName="test-curl-7589-e2e17e"
Sep 09 20:30:06.009324 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.009313    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b9f9587-5be9-4ba4-b90e-54d55bc354a1" containerName="test-curl-7589-e2e17e"
Sep 09 20:30:06.012534 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.012512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.015203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.015182    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-812760-openshift-default-dockercfg-7f2qx\""
Sep 09 20:30:06.025880 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.025848    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"]
Sep 09 20:30:06.122253 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.122213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d67d76-d49b-451c-93ab-a9e53afda094-istiod-ca-cert\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.122425 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.122264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-istio-envoy\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.122425 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.122321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d67d76-d49b-451c-93ab-a9e53afda094-istio-podinfo\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.122425 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.122369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d67d76-d49b-451c-93ab-a9e53afda094-istio-token\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.122425 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.122418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxn2g\" (UniqueName: \"kubernetes.io/projected/69d67d76-d49b-451c-93ab-a9e53afda094-kube-api-access-wxn2g\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.122620 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.122502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-workload-socket\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.122620 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.122527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-workload-certs\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.122620 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.122557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-credential-socket\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.122620 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.122603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-istio-data\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.217977 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.217936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:06.223951 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.223916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-istio-envoy\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.223951 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.223964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d67d76-d49b-451c-93ab-a9e53afda094-istio-podinfo\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.224162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.223985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d67d76-d49b-451c-93ab-a9e53afda094-istio-token\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.224162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.224011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxn2g\" (UniqueName: \"kubernetes.io/projected/69d67d76-d49b-451c-93ab-a9e53afda094-kube-api-access-wxn2g\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.224162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.224077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-workload-socket\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.224162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.224101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-workload-certs\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.224162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.224125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-credential-socket\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.224162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.224154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-istio-data\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.224489 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.224206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d67d76-d49b-451c-93ab-a9e53afda094-istiod-ca-cert\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.224591 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.224550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-workload-certs\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.224664 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.224615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-credential-socket\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.224703 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.224683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-workload-socket\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.224734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.224695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-istio-data\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.224989 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.224966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d67d76-d49b-451c-93ab-a9e53afda094-istiod-ca-cert\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.226472 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.226449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-istio-envoy\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.226564 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.226524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d67d76-d49b-451c-93ab-a9e53afda094-istio-podinfo\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.232863 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.232828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxn2g\" (UniqueName: \"kubernetes.io/projected/69d67d76-d49b-451c-93ab-a9e53afda094-kube-api-access-wxn2g\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.233045 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.233028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d67d76-d49b-451c-93ab-a9e53afda094-istio-token\") pod \"e2e-autores-812760-openshift-default-5cd46b49c8-725xh\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") " pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.325288 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.325198    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:06.567031 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.566998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:06.582988 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.582895    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7589-e2e17e" podUID="1b9f9587-5be9-4ba4-b90e-54d55bc354a1" containerName="test-curl-7589-e2e17e" containerID="cri-o://601f4d3358fd44134e94774f68138c286ec4dcf90a2d14cc773c80a49d8d6b8b" gracePeriod=2
Sep 09 20:30:06.609912 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.609879    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"]
Sep 09 20:30:06.613429 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:30:06.613396    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d67d76_d49b_451c_93ab_a9e53afda094.slice/crio-f5f30243ad27bed66d168d1358b573b84370718f8c40f709b1c85aaf1aa4a165 WatchSource:0}: Error finding container f5f30243ad27bed66d168d1358b573b84370718f8c40f709b1c85aaf1aa4a165: Status 404 returned error can't find the container with id f5f30243ad27bed66d168d1358b573b84370718f8c40f709b1c85aaf1aa4a165
Sep 09 20:30:06.615949 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.615909    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:06.616034 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.616003    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:06.616080 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.616044    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:06.703709 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:06.703679    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816430-f8b4k"
Sep 09 20:30:07.025958 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:07.025886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:07.126098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:07.126060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:07.519359 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:07.519315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:07.589058 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:07.588961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" event={"ID":"69d67d76-d49b-451c-93ab-a9e53afda094","Type":"ContainerStarted","Data":"f7e299e6246c0c0adf5bd2ee7b4d21ac312720ef122c69639d42b42f70f50e30"}
Sep 09 20:30:07.589236 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:07.589057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" event={"ID":"69d67d76-d49b-451c-93ab-a9e53afda094","Type":"ContainerStarted","Data":"f5f30243ad27bed66d168d1358b573b84370718f8c40f709b1c85aaf1aa4a165"}
Sep 09 20:30:07.590304 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:07.590284    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816430-f8b4k"
Sep 09 20:30:07.590421 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:07.590282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816430-f8b4k" event={"ID":"b0e24d8b-8b52-4b62-8419-a82b3b7cdf16","Type":"ContainerDied","Data":"561bf095d7e18215d90b096180bab6e66de4ad82b3c045c1766dcd2a9d1be2b2"}
Sep 09 20:30:07.590421 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:07.590403    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="561bf095d7e18215d90b096180bab6e66de4ad82b3c045c1766dcd2a9d1be2b2"
Sep 09 20:30:07.612417 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:07.612365    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" podStartSLOduration=2.612350282 podStartE2EDuration="2.612350282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:07.611151542 +0000 UTC m=+814.850253844" watchObservedRunningTime="2026-09-09 20:30:07.612350282 +0000 UTC m=+814.851452582"
Sep 09 20:30:07.617529 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:07.617493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:07.689561 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:07.689526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:08.325836 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:08.325798    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:08.330841 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:08.330812    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:08.595481 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:08.595453    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:08.596621 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:08.596602    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:08.799847 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:08.799822    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7589-e2e17e"
Sep 09 20:30:08.962746 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:08.962706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78szs\" (UniqueName: \"kubernetes.io/projected/1b9f9587-5be9-4ba4-b90e-54d55bc354a1-kube-api-access-78szs\") pod \"1b9f9587-5be9-4ba4-b90e-54d55bc354a1\" (UID: \"1b9f9587-5be9-4ba4-b90e-54d55bc354a1\") "
Sep 09 20:30:08.965839 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:08.965796    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9f9587-5be9-4ba4-b90e-54d55bc354a1-kube-api-access-78szs" (OuterVolumeSpecName: "kube-api-access-78szs") pod "1b9f9587-5be9-4ba4-b90e-54d55bc354a1" (UID: "1b9f9587-5be9-4ba4-b90e-54d55bc354a1"). InnerVolumeSpecName "kube-api-access-78szs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:30:08.993265 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:08.993223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:09.063932 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:09.063888    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78szs\" (UniqueName: \"kubernetes.io/projected/1b9f9587-5be9-4ba4-b90e-54d55bc354a1-kube-api-access-78szs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:30:09.313793 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:09.313710    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b9f9587-5be9-4ba4-b90e-54d55bc354a1" path="/var/lib/kubelet/pods/1b9f9587-5be9-4ba4-b90e-54d55bc354a1/volumes"
Sep 09 20:30:09.590853 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:09.590757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:09.600854 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:09.600822    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b9f9587-5be9-4ba4-b90e-54d55bc354a1" containerID="601f4d3358fd44134e94774f68138c286ec4dcf90a2d14cc773c80a49d8d6b8b" exitCode=137
Sep 09 20:30:09.601264 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:09.600869    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7589-e2e17e"
Sep 09 20:30:09.601264 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:09.600904    2563 scope.go:117] "RemoveContainer" containerID="601f4d3358fd44134e94774f68138c286ec4dcf90a2d14cc773c80a49d8d6b8b"
Sep 09 20:30:09.610434 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:09.610415    2563 scope.go:117] "RemoveContainer" containerID="601f4d3358fd44134e94774f68138c286ec4dcf90a2d14cc773c80a49d8d6b8b"
Sep 09 20:30:09.610715 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:30:09.610693    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601f4d3358fd44134e94774f68138c286ec4dcf90a2d14cc773c80a49d8d6b8b\": container with ID starting with 601f4d3358fd44134e94774f68138c286ec4dcf90a2d14cc773c80a49d8d6b8b not found: ID does not exist" containerID="601f4d3358fd44134e94774f68138c286ec4dcf90a2d14cc773c80a49d8d6b8b"
Sep 09 20:30:09.610765 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:09.610726    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601f4d3358fd44134e94774f68138c286ec4dcf90a2d14cc773c80a49d8d6b8b"} err="failed to get container status \"601f4d3358fd44134e94774f68138c286ec4dcf90a2d14cc773c80a49d8d6b8b\": rpc error: code = NotFound desc = could not find container \"601f4d3358fd44134e94774f68138c286ec4dcf90a2d14cc773c80a49d8d6b8b\": container with ID starting with 601f4d3358fd44134e94774f68138c286ec4dcf90a2d14cc773c80a49d8d6b8b not found: ID does not exist"
Sep 09 20:30:09.683468 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:09.683433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:09.887253 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:09.887124    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:10.601514 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:10.601473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:11.493771 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:11.493734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:11.579118 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:11.579079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:11.680036 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:11.680003    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:11.989718 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:11.989686    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:12.093284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.093248    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"]
Sep 09 20:30:12.093611 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.093597    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0e24d8b-8b52-4b62-8419-a82b3b7cdf16" containerName="cleanup"
Sep 09 20:30:12.093659 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.093612    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0e24d8b-8b52-4b62-8419-a82b3b7cdf16" containerName="cleanup"
Sep 09 20:30:12.093691 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.093660    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0e24d8b-8b52-4b62-8419-a82b3b7cdf16" containerName="cleanup"
Sep 09 20:30:12.097824 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.097804    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"
Sep 09 20:30:12.100479 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.100455    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-812760-dockercfg-zr6tq\""
Sep 09 20:30:12.100601 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.100514    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-812760\""
Sep 09 20:30:12.107520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.107500    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"]
Sep 09 20:30:12.172486 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.172312    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:12.193282 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.193244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk\" (UID: \"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"
Sep 09 20:30:12.193282 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.193287    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s56kd\" (UniqueName: \"kubernetes.io/projected/03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b-kube-api-access-s56kd\") pod \"payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk\" (UID: \"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"
Sep 09 20:30:12.294667 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.294628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk\" (UID: \"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"
Sep 09 20:30:12.294667 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.294670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s56kd\" (UniqueName: \"kubernetes.io/projected/03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b-kube-api-access-s56kd\") pod \"payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk\" (UID: \"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"
Sep 09 20:30:12.295360 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.295336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk\" (UID: \"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"
Sep 09 20:30:12.303036 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.303009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s56kd\" (UniqueName: \"kubernetes.io/projected/03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b-kube-api-access-s56kd\") pod \"payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk\" (UID: \"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"
Sep 09 20:30:12.409392 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.409350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"
Sep 09 20:30:12.511887 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.511802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:12.705798 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:12.705771    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"]
Sep 09 20:30:12.708166 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:30:12.708129    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b5e00e_f738_4b8b_9bc4_9d62cd37fe4b.slice/crio-16911e98b2f58dcd19d69c75cc5e0f617ff7b1c48abffabf0fe2eff17dabe8d1 WatchSource:0}: Error finding container 16911e98b2f58dcd19d69c75cc5e0f617ff7b1c48abffabf0fe2eff17dabe8d1: Status 404 returned error can't find the container with id 16911e98b2f58dcd19d69c75cc5e0f617ff7b1c48abffabf0fe2eff17dabe8d1
Sep 09 20:30:13.185533 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:13.185443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:13.442040 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:13.441948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:13.623179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:13.623136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk" event={"ID":"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b","Type":"ContainerStarted","Data":"16911e98b2f58dcd19d69c75cc5e0f617ff7b1c48abffabf0fe2eff17dabe8d1"}
Sep 09 20:30:13.646460 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:13.646426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:13.922652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:13.921765    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"]
Sep 09 20:30:13.926100 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:13.926068    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:13.929206 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:13.929181    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4defd2-openshift-default-dockercfg-zdwcv\""
Sep 09 20:30:13.938823 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:13.938796    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"]
Sep 09 20:30:13.967952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:13.967893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:14.010921 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.010877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-istio-envoy\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.011119 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.010931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f688849-f103-4589-8258-8e83a3015197-istio-token\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.011119 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.010973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-workload-socket\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.011119 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.011057    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-workload-certs\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.011119 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.011095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k47ks\" (UniqueName: \"kubernetes.io/projected/1f688849-f103-4589-8258-8e83a3015197-kube-api-access-k47ks\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.011323 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.011132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-credential-socket\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.011323 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.011161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f688849-f103-4589-8258-8e83a3015197-istio-podinfo\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.011323 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.011217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f688849-f103-4589-8258-8e83a3015197-istiod-ca-cert\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.011323 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.011245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-istio-data\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.064564 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.064519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:14.112702 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.112662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f688849-f103-4589-8258-8e83a3015197-istiod-ca-cert\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.112702 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.112706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-istio-data\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.112940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.112746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-istio-envoy\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.112940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.112775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f688849-f103-4589-8258-8e83a3015197-istio-token\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.112940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.112811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-workload-socket\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.112940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.112868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-workload-certs\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.112940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.112903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k47ks\" (UniqueName: \"kubernetes.io/projected/1f688849-f103-4589-8258-8e83a3015197-kube-api-access-k47ks\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.113178 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.112955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-credential-socket\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.113178 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.112982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f688849-f103-4589-8258-8e83a3015197-istio-podinfo\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.113178 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.113159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-istio-data\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.113437 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.113413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-workload-certs\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.113521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.113499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f688849-f103-4589-8258-8e83a3015197-istiod-ca-cert\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.113783 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.113756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-workload-socket\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.113865 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.113797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-credential-socket\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.116685 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.116660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f688849-f103-4589-8258-8e83a3015197-istio-podinfo\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.116885 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.116864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-istio-envoy\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.137102 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.137025    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f688849-f103-4589-8258-8e83a3015197-istio-token\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.140791 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.140739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k47ks\" (UniqueName: \"kubernetes.io/projected/1f688849-f103-4589-8258-8e83a3015197-kube-api-access-k47ks\") pod \"e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") " pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.182926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.182802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:14.245399 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.245303    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:14.296159 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.296120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:14.387598 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.387523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:14.488297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.488228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:14.539111 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.539075    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"]
Sep 09 20:30:14.580791 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.580748    2563 patch_prober.go:28] interesting pod/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 20:30:14.580979 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:14.580841    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:30:15.056390 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:30:15.056343    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f688849_f103_4589_8258_8e83a3015197.slice/crio-cd162f8439812a069c3d8cac6bd66f8c453c1e29f919e9d37ddcfadd2a5df708 WatchSource:0}: Error finding container cd162f8439812a069c3d8cac6bd66f8c453c1e29f919e9d37ddcfadd2a5df708: Status 404 returned error can't find the container with id cd162f8439812a069c3d8cac6bd66f8c453c1e29f919e9d37ddcfadd2a5df708
Sep 09 20:30:15.200319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:15.200274    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:30:15.375798 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:15.375713    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:15.375946 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:15.375801    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:15.375946 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:15.375841    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:15.635600 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:15.635484    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" event={"ID":"1f688849-f103-4589-8258-8e83a3015197","Type":"ContainerStarted","Data":"30cf8904de98fcfe3bd5e16bc0e4a9e23b035c9ef090b36e4cd32791f9d218cb"}
Sep 09 20:30:15.635600 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:15.635562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" event={"ID":"1f688849-f103-4589-8258-8e83a3015197","Type":"ContainerStarted","Data":"cd162f8439812a069c3d8cac6bd66f8c453c1e29f919e9d37ddcfadd2a5df708"}
Sep 09 20:30:15.637350 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:15.637321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk" event={"ID":"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b","Type":"ContainerStarted","Data":"481ca5107d3197f2e13e155c9e92236e6c9326823b9379903d4a96151d4b13fa"}
Sep 09 20:30:15.637483 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:15.637452    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"
Sep 09 20:30:15.658342 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:15.658288    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" podStartSLOduration=2.6582650020000003 podStartE2EDuration="2.658265002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:15.653645478 +0000 UTC m=+822.892747780" watchObservedRunningTime="2026-09-09 20:30:15.658265002 +0000 UTC m=+822.897367298"
Sep 09 20:30:15.672088 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:15.672027    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk" podStartSLOduration=0.949443206 podStartE2EDuration="3.672005528s" totalImagesPullingTime="2.722562322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:15.66934147 +0000 UTC m=+822.908443763" watchObservedRunningTime="2026-09-09 20:30:15.672005528 +0000 UTC m=+822.911107829"
Sep 09 20:30:16.246592 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:16.246539    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:16.252300 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:16.252273    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:16.642978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:16.642938    2563 generic.go:356] "Generic (PLEG): container finished" podID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerID="b798fa556afa4f3de4628b0a36fa674b416b896bedb60cb7fd71569bf37fffce" exitCode=0
Sep 09 20:30:16.643163 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:16.643003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" event={"ID":"1314a2b3-c135-4998-99b5-76a9b81370c2","Type":"ContainerDied","Data":"b798fa556afa4f3de4628b0a36fa674b416b896bedb60cb7fd71569bf37fffce"}
Sep 09 20:30:16.643727 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:16.643636    2563 scope.go:117] "RemoveContainer" containerID="b798fa556afa4f3de4628b0a36fa674b416b896bedb60cb7fd71569bf37fffce"
Sep 09 20:30:16.643727 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:16.643649    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:16.644545 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:16.644384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:16.644846 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:16.644816    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:16.644963 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:16.644886    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:16.644963 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:16.644917    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:17.139792 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:17.139753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:17.652397 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:17.652351    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" event={"ID":"1314a2b3-c135-4998-99b5-76a9b81370c2","Type":"ContainerStarted","Data":"4ddff32e89e1316fa334888e3ac654c67a84a974cd5366d7aa46d586927f5a1e"}
Sep 09 20:30:17.689680 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:17.689646    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:30:17.690062 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:17.689970    2563 patch_prober.go:28] interesting pod/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 09 20:30:17.690216 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:17.690195    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 09 20:30:17.718120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:17.718080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:17.806528 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:17.806501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:18.689168 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:18.689132    2563 patch_prober.go:28] interesting pod/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 09 20:30:18.689776 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:18.689742    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 09 20:30:18.704540 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:18.704492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:19.077417 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:19.077374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:19.689505 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:19.689464    2563 patch_prober.go:28] interesting pod/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 09 20:30:19.690147 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:19.690114    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 09 20:30:19.811322 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:19.811287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:20.421997 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:20.421968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:20.523433 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:20.523395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:20.622818 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:20.622780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:20.689485 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:20.689386    2563 patch_prober.go:28] interesting pod/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 09 20:30:20.689485 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:20.689460    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 09 20:30:20.756601 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:20.756552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:21.690333 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:21.689847    2563 patch_prober.go:28] interesting pod/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 09 20:30:21.690333 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:21.689916    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 09 20:30:21.804141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:21.804060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:21.878107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:21.875111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:22.340793 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.336079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:22.437821 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.437789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:22.441279 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.441254    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"]
Sep 09 20:30:22.484594 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.483897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"]
Sep 09 20:30:22.484594 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.484041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"
Sep 09 20:30:22.497911 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.497880    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4defd2\""
Sep 09 20:30:22.515847 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.515816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:22.612855 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.612763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ddea5f1-7cb2-4df5-ae87-1f900be04a67-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5\" (UID: \"0ddea5f1-7cb2-4df5-ae87-1f900be04a67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"
Sep 09 20:30:22.613015 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.612924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2p78\" (UniqueName: \"kubernetes.io/projected/0ddea5f1-7cb2-4df5-ae87-1f900be04a67-kube-api-access-p2p78\") pod \"payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5\" (UID: \"0ddea5f1-7cb2-4df5-ae87-1f900be04a67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"
Sep 09 20:30:22.688935 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.688898    2563 patch_prober.go:28] interesting pod/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 09 20:30:22.689078 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.688969    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 09 20:30:22.713810 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.713776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2p78\" (UniqueName: \"kubernetes.io/projected/0ddea5f1-7cb2-4df5-ae87-1f900be04a67-kube-api-access-p2p78\") pod \"payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5\" (UID: \"0ddea5f1-7cb2-4df5-ae87-1f900be04a67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"
Sep 09 20:30:22.714118 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.713850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ddea5f1-7cb2-4df5-ae87-1f900be04a67-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5\" (UID: \"0ddea5f1-7cb2-4df5-ae87-1f900be04a67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"
Sep 09 20:30:22.714481 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.714462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ddea5f1-7cb2-4df5-ae87-1f900be04a67-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5\" (UID: \"0ddea5f1-7cb2-4df5-ae87-1f900be04a67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"
Sep 09 20:30:22.722212 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.722177    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2p78\" (UniqueName: \"kubernetes.io/projected/0ddea5f1-7cb2-4df5-ae87-1f900be04a67-kube-api-access-p2p78\") pod \"payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5\" (UID: \"0ddea5f1-7cb2-4df5-ae87-1f900be04a67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"
Sep 09 20:30:22.814595 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.814533    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"
Sep 09 20:30:22.955235 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:22.955051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:23.088691 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:23.088631    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"]
Sep 09 20:30:23.091024 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:30:23.090991    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddea5f1_7cb2_4df5_ae87_1f900be04a67.slice/crio-59c6a9f5097bddce6f919a0605cf30e650765a42651c9c45a7e553bcc95561e3 WatchSource:0}: Error finding container 59c6a9f5097bddce6f919a0605cf30e650765a42651c9c45a7e553bcc95561e3: Status 404 returned error can't find the container with id 59c6a9f5097bddce6f919a0605cf30e650765a42651c9c45a7e553bcc95561e3
Sep 09 20:30:23.692519 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:23.692477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5" event={"ID":"0ddea5f1-7cb2-4df5-ae87-1f900be04a67","Type":"ContainerStarted","Data":"5ebd9d5e04d0d2bb2b80f241fe29767d26cd8d4a868ac585bc402f6d2f719b00"}
Sep 09 20:30:23.692737 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:23.692532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5" event={"ID":"0ddea5f1-7cb2-4df5-ae87-1f900be04a67","Type":"ContainerStarted","Data":"59c6a9f5097bddce6f919a0605cf30e650765a42651c9c45a7e553bcc95561e3"}
Sep 09 20:30:23.692737 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:23.692630    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"
Sep 09 20:30:23.694300 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:23.694274    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:30:23.694563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:23.694541    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:30:23.695356 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:23.695339    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:30:23.709642 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:23.709593    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5" podStartSLOduration=1.7095668069999999 podStartE2EDuration="1.709566807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:23.707725351 +0000 UTC m=+830.946827657" watchObservedRunningTime="2026-09-09 20:30:23.709566807 +0000 UTC m=+830.948669160"
Sep 09 20:30:23.799960 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:23.799920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:24.310203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:24.310165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:24.589789 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:24.589706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:24.637791 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:24.637749    2563 patch_prober.go:28] interesting pod/e2e-autores-812760-openshift-default-5cd46b49c8-725xh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 20:30:24.637955 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:24.637805    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:30:25.393871 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:25.393832    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:25.703949 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:25.703868    2563 generic.go:356] "Generic (PLEG): container finished" podID="69d67d76-d49b-451c-93ab-a9e53afda094" containerID="f7e299e6246c0c0adf5bd2ee7b4d21ac312720ef122c69639d42b42f70f50e30" exitCode=0
Sep 09 20:30:25.704122 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:25.703941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" event={"ID":"69d67d76-d49b-451c-93ab-a9e53afda094","Type":"ContainerDied","Data":"f7e299e6246c0c0adf5bd2ee7b4d21ac312720ef122c69639d42b42f70f50e30"}
Sep 09 20:30:25.704428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:25.704411    2563 scope.go:117] "RemoveContainer" containerID="f7e299e6246c0c0adf5bd2ee7b4d21ac312720ef122c69639d42b42f70f50e30"
Sep 09 20:30:25.705600 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:25.705549    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:25.705712 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:25.705631    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:25.705712 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:25.705670    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:26.133887 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.133851    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg"]
Sep 09 20:30:26.136148 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.136133    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg"
Sep 09 20:30:26.138484 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.138453    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:30:26.138484 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.138453    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:30:26.143988 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.143823    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg"]
Sep 09 20:30:26.147258 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.147234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a4cb7a-e612-4558-8726-5733e2553235-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg\" (UID: \"86a4cb7a-e612-4558-8726-5733e2553235\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg"
Sep 09 20:30:26.193772 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.193708    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:26.247870 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.247830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a4cb7a-e612-4558-8726-5733e2553235-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg\" (UID: \"86a4cb7a-e612-4558-8726-5733e2553235\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg"
Sep 09 20:30:26.248414 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.248393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a4cb7a-e612-4558-8726-5733e2553235-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg\" (UID: \"86a4cb7a-e612-4558-8726-5733e2553235\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg"
Sep 09 20:30:26.448156 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.448057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg"
Sep 09 20:30:26.667596 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.662895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"
Sep 09 20:30:26.713194 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.713161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" event={"ID":"69d67d76-d49b-451c-93ab-a9e53afda094","Type":"ContainerStarted","Data":"8c236f9e2dcda96c18a7c453b7836133cc6485e4a830d600c9fd202a0f483a50"}
Sep 09 20:30:26.740983 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.740950    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg"]
Sep 09 20:30:26.774296 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:30:26.774258    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a4cb7a_e612_4558_8726_5733e2553235.slice/crio-13e664822a190ed20913d640087683a4e282a8d1b5907271834360f115103279 WatchSource:0}: Error finding container 13e664822a190ed20913d640087683a4e282a8d1b5907271834360f115103279: Status 404 returned error can't find the container with id 13e664822a190ed20913d640087683a4e282a8d1b5907271834360f115103279
Sep 09 20:30:26.988747 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:26.988660    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:27.326254 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:27.326174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:27.326482 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:27.326461    2563 patch_prober.go:28] interesting pod/e2e-autores-812760-openshift-default-5cd46b49c8-725xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 09 20:30:27.326531 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:27.326508    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 09 20:30:27.604881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:27.604787    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:27.706466 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:27.706431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:27.719644 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:27.719613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg" event={"ID":"86a4cb7a-e612-4558-8726-5733e2553235","Type":"ContainerStarted","Data":"c31b9eea1c51a192c069f80d0665561733afe4355f0a90854f781408f1874def"}
Sep 09 20:30:27.719778 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:27.719661    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg" event={"ID":"86a4cb7a-e612-4558-8726-5733e2553235","Type":"ContainerStarted","Data":"13e664822a190ed20913d640087683a4e282a8d1b5907271834360f115103279"}
Sep 09 20:30:27.740777 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:27.740731    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg" podStartSLOduration=1.740716555 podStartE2EDuration="1.740716555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:27.740654272 +0000 UTC m=+834.979756572" watchObservedRunningTime="2026-09-09 20:30:27.740716555 +0000 UTC m=+834.979818862"
Sep 09 20:30:27.803945 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:27.803914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:28.191113 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:28.190956    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:28.286257 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:28.286181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:28.325796 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:28.325758    2563 patch_prober.go:28] interesting pod/e2e-autores-812760-openshift-default-5cd46b49c8-725xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 09 20:30:28.325986 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:28.325836    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 09 20:30:28.726162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:28.726127    2563 generic.go:356] "Generic (PLEG): container finished" podID="86a4cb7a-e612-4558-8726-5733e2553235" containerID="c31b9eea1c51a192c069f80d0665561733afe4355f0a90854f781408f1874def" exitCode=0
Sep 09 20:30:28.726649 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:28.726230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg" event={"ID":"86a4cb7a-e612-4558-8726-5733e2553235","Type":"ContainerDied","Data":"c31b9eea1c51a192c069f80d0665561733afe4355f0a90854f781408f1874def"}
Sep 09 20:30:29.102166 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:29.102071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:29.326275 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:29.326243    2563 patch_prober.go:28] interesting pod/e2e-autores-812760-openshift-default-5cd46b49c8-725xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 09 20:30:29.326446 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:29.326293    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 09 20:30:29.967893 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:29.967560    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg"
Sep 09 20:30:30.004371 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:30.004272    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a4cb7a-e612-4558-8726-5733e2553235-maas-api-service-ca\") pod \"86a4cb7a-e612-4558-8726-5733e2553235\" (UID: \"86a4cb7a-e612-4558-8726-5733e2553235\") "
Sep 09 20:30:30.005216 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:30.005154    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a4cb7a-e612-4558-8726-5733e2553235-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86a4cb7a-e612-4558-8726-5733e2553235" (UID: "86a4cb7a-e612-4558-8726-5733e2553235"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:30:30.097190 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:30.097140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:30.106201 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:30.106169    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a4cb7a-e612-4558-8726-5733e2553235-maas-api-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:30:30.326680 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:30.326484    2563 patch_prober.go:28] interesting pod/e2e-autores-812760-openshift-default-5cd46b49c8-725xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 09 20:30:30.326680 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:30.326546    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 09 20:30:30.736602 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:30.736549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg" event={"ID":"86a4cb7a-e612-4558-8726-5733e2553235","Type":"ContainerDied","Data":"13e664822a190ed20913d640087683a4e282a8d1b5907271834360f115103279"}
Sep 09 20:30:30.736748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:30.736616    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13e664822a190ed20913d640087683a4e282a8d1b5907271834360f115103279"
Sep 09 20:30:30.736748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:30.736664    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg"
Sep 09 20:30:30.845854 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:30.845812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:31.326172 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:31.325836    2563 patch_prober.go:28] interesting pod/e2e-autores-812760-openshift-default-5cd46b49c8-725xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 09 20:30:31.326172 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:31.325893    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 09 20:30:31.782865 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:31.782809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:32.306068 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:32.306029    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:32.329189 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:32.329160    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:32.329555 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:32.329449    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:32.330090 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:32.330070    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:30:32.514462 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:32.514424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:33.306348 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:33.306300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:34.009776 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:34.009739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:34.700510 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:34.700479    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"
Sep 09 20:30:34.810699 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:34.810658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:34.901426 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:34.901379    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:35.000471 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:35.000424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:35.631905 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:35.631867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:36.248149 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.248113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:36.316453 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.316408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:36.380088 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.380053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:36.433451 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.433419    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg"]
Sep 09 20:30:36.436327 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.436301    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87ac1d4f-6b4kg"]
Sep 09 20:30:36.472031 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.471993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:36.535792 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.535726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:36.651147 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.651114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:36.776786 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.776705    2563 generic.go:356] "Generic (PLEG): container finished" podID="1f688849-f103-4589-8258-8e83a3015197" containerID="30cf8904de98fcfe3bd5e16bc0e4a9e23b035c9ef090b36e4cd32791f9d218cb" exitCode=0
Sep 09 20:30:36.776786 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.776779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" event={"ID":"1f688849-f103-4589-8258-8e83a3015197","Type":"ContainerDied","Data":"30cf8904de98fcfe3bd5e16bc0e4a9e23b035c9ef090b36e4cd32791f9d218cb"}
Sep 09 20:30:36.777163 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.777149    2563 scope.go:117] "RemoveContainer" containerID="30cf8904de98fcfe3bd5e16bc0e4a9e23b035c9ef090b36e4cd32791f9d218cb"
Sep 09 20:30:36.778174 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.778144    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:36.778274 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.778225    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:36.778334 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.778270    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:36.890199 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:36.890164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:37.097373 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.097277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:37.195121 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.195078    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:37.318223 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.318187    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86a4cb7a-e612-4558-8726-5733e2553235" path="/var/lib/kubelet/pods/86a4cb7a-e612-4558-8726-5733e2553235/volumes"
Sep 09 20:30:37.358878 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.358788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:37.695235 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.695142    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"]
Sep 09 20:30:37.695674 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.695656    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a4cb7a-e612-4558-8726-5733e2553235" containerName="revoke-keys"
Sep 09 20:30:37.695727 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.695675    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a4cb7a-e612-4558-8726-5733e2553235" containerName="revoke-keys"
Sep 09 20:30:37.695767 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.695742    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="86a4cb7a-e612-4558-8726-5733e2553235" containerName="revoke-keys"
Sep 09 20:30:37.697954 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.697937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.700855 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.700834    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-25e8b2a0-gw-openshift-default-dockercfg-s5vmm\""
Sep 09 20:30:37.712635 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.712610    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"]
Sep 09 20:30:37.755336 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.755299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:37.776961 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.775960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-workload-socket\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.776961 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.776009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-workload-certs\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.776961 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.776040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnp55\" (UniqueName: \"kubernetes.io/projected/4cbb079f-0902-45d2-8d3d-6faee0a084e4-kube-api-access-tnp55\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.776961 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.776067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-data\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.776961 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.776098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-podinfo\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.776961 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.776141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-credential-socket\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.776961 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.776212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-envoy\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.776961 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.776235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-token\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.776961 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.776270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istiod-ca-cert\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.786693 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.786659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" event={"ID":"1f688849-f103-4589-8258-8e83a3015197","Type":"ContainerStarted","Data":"33ccb910b7e0611612eb40195633c732d071dae0e900edc6f3928efcdb5bbc18"}
Sep 09 20:30:37.877884 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.877845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-workload-socket\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.877884 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.877882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-workload-certs\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.878137 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.877904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnp55\" (UniqueName: \"kubernetes.io/projected/4cbb079f-0902-45d2-8d3d-6faee0a084e4-kube-api-access-tnp55\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.878137 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.877927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-data\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.878137 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.877964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-podinfo\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.878137 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.878007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-credential-socket\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.878137 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.878076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-envoy\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.878137 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.878099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-token\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.878428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.878165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istiod-ca-cert\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.878428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.878393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-workload-certs\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.878526 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.878426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-workload-socket\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.878885 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.878686    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-credential-socket\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.878885 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.878799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-data\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.879042 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.878897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istiod-ca-cert\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.880594 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.880556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-envoy\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.880794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.880778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-podinfo\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.890558 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.890536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnp55\" (UniqueName: \"kubernetes.io/projected/4cbb079f-0902-45d2-8d3d-6faee0a084e4-kube-api-access-tnp55\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.890687 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.890563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-token\") pod \"e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") " pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:37.895603 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:37.895540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:38.009179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.009138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:38.107684 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.106991    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:30:38.195963 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.195931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:38.249688 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.248956    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:38.249688 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.249124    2563 patch_prober.go:28] interesting pod/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 09 20:30:38.249688 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.249172    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 09 20:30:38.318813 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.318787    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"]
Sep 09 20:30:38.320720 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:30:38.320686    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbb079f_0902_45d2_8d3d_6faee0a084e4.slice/crio-1dde67a7b3eb63f88815646d3576ce64771a4e677cb8c5601457b1746c2f42f6 WatchSource:0}: Error finding container 1dde67a7b3eb63f88815646d3576ce64771a4e677cb8c5601457b1746c2f42f6: Status 404 returned error can't find the container with id 1dde67a7b3eb63f88815646d3576ce64771a4e677cb8c5601457b1746c2f42f6
Sep 09 20:30:38.322974 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.322943    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:38.323041 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.323009    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:38.323081 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.323039    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:30:38.794653 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.794613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5" event={"ID":"4cbb079f-0902-45d2-8d3d-6faee0a084e4","Type":"ContainerStarted","Data":"9df18472e6c3b42c6154a562eb5fb778cd210a757b4bc780c79a9af85d8ce566"}
Sep 09 20:30:38.795045 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.794673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5" event={"ID":"4cbb079f-0902-45d2-8d3d-6faee0a084e4","Type":"ContainerStarted","Data":"1dde67a7b3eb63f88815646d3576ce64771a4e677cb8c5601457b1746c2f42f6"}
Sep 09 20:30:38.815133 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.815081    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5" podStartSLOduration=1.815063492 podStartE2EDuration="1.815063492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:38.814174358 +0000 UTC m=+846.053276660" watchObservedRunningTime="2026-09-09 20:30:38.815063492 +0000 UTC m=+846.054165854"
Sep 09 20:30:38.878296 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:38.878260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:39.010271 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:39.010235    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:39.011842 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:39.011814    2563 patch_prober.go:28] interesting pod/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 09 20:30:39.011960 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:39.011867    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5" podUID="4cbb079f-0902-45d2-8d3d-6faee0a084e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 09 20:30:39.246388 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:39.246344    2563 patch_prober.go:28] interesting pod/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 09 20:30:39.246608 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:39.246423    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 09 20:30:39.802801 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:39.802763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:40.014288 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:40.014253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:40.246310 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:40.246275    2563 patch_prober.go:28] interesting pod/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 09 20:30:40.246642 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:40.246612    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 09 20:30:40.552482 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:40.552389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:40.649381 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:40.649350    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:40.770249 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:40.770204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:40.804709 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:40.804613    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:40.806080 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:40.806061    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:30:41.229517 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.229484    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"]
Sep 09 20:30:41.231874 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.231853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.234037 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.234016    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-836c5d71-kserve-self-signed-certs\""
Sep 09 20:30:41.243596 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.243537    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"]
Sep 09 20:30:41.246812 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.246779    2563 patch_prober.go:28] interesting pod/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 09 20:30:41.246916 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.246843    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 09 20:30:41.309742 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.309697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-dshm\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.309898 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.309745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-home\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.309898 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.309816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-tmp-dir\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.310001 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.309940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggh9r\" (UniqueName: \"kubernetes.io/projected/290ca516-9aa2-48ed-924b-b812e2656b11-kube-api-access-ggh9r\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.310040 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.309997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/290ca516-9aa2-48ed-924b-b812e2656b11-tls-certs\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.310078 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.310041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-model-cache\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.411001 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.410955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-tmp-dir\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.411224 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.411165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggh9r\" (UniqueName: \"kubernetes.io/projected/290ca516-9aa2-48ed-924b-b812e2656b11-kube-api-access-ggh9r\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.411305 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.411234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/290ca516-9aa2-48ed-924b-b812e2656b11-tls-certs\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.411305 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.411274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-model-cache\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.411400 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.411309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-dshm\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.411400 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.411329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-tmp-dir\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.411400 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.411339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-home\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.411678 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.411653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-home\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.411791 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.411770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-model-cache\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.413541 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.413524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-dshm\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.413644 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.413628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/290ca516-9aa2-48ed-924b-b812e2656b11-tls-certs\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.419347 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.419326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggh9r\" (UniqueName: \"kubernetes.io/projected/290ca516-9aa2-48ed-924b-b812e2656b11-kube-api-access-ggh9r\") pod \"e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") " pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.479715 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.479623    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:41.541475 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.541434    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:41.624610 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.624036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:41.834207 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:41.834125    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"]
Sep 09 20:30:41.837536 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:30:41.837500    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod290ca516_9aa2_48ed_924b_b812e2656b11.slice/crio-5b288d2e9396f4526ebe088c5847abf18de92c1fbdac32348f02b9b591da34dd WatchSource:0}: Error finding container 5b288d2e9396f4526ebe088c5847abf18de92c1fbdac32348f02b9b591da34dd: Status 404 returned error can't find the container with id 5b288d2e9396f4526ebe088c5847abf18de92c1fbdac32348f02b9b591da34dd
Sep 09 20:30:42.246807 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:42.246762    2563 patch_prober.go:28] interesting pod/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 09 20:30:42.247015 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:42.246839    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 09 20:30:42.379473 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:42.379434    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:42.816164 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:42.816131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk" event={"ID":"290ca516-9aa2-48ed-924b-b812e2656b11","Type":"ContainerStarted","Data":"e99566f481d844887a176b0380489d2075b6bdb8d3312617fbb8e23aaa46928e"}
Sep 09 20:30:42.816346 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:42.816176    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk" event={"ID":"290ca516-9aa2-48ed-924b-b812e2656b11","Type":"ContainerStarted","Data":"5b288d2e9396f4526ebe088c5847abf18de92c1fbdac32348f02b9b591da34dd"}
Sep 09 20:30:42.831317 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:42.831261    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk" podStartSLOduration=1.831243836 podStartE2EDuration="1.831243836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:42.830989106 +0000 UTC m=+850.070091407" watchObservedRunningTime="2026-09-09 20:30:42.831243836 +0000 UTC m=+850.070346136"
Sep 09 20:30:43.135172 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.135064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:43.238775 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.238734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:43.251596 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.246926    2563 patch_prober.go:28] interesting pod/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 09 20:30:43.251596 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.246986    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 09 20:30:43.336559 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.336519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:43.602521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.602475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:43.705630 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.705565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:43.738497 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.738454    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"]
Sep 09 20:30:43.741175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.741155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"
Sep 09 20:30:43.743517 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.743489    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-25e8b2a0\""
Sep 09 20:30:43.743671 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.743622    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-25e8b2a0-dockercfg-dszdb\""
Sep 09 20:30:43.754930 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.754903    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"]
Sep 09 20:30:43.840594 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.838453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgh4k\" (UniqueName: \"kubernetes.io/projected/b4adc0d7-6356-4632-8a7e-f0955173287e-kube-api-access-zgh4k\") pod \"payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5\" (UID: \"b4adc0d7-6356-4632-8a7e-f0955173287e\") " pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"
Sep 09 20:30:43.840594 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.838511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4adc0d7-6356-4632-8a7e-f0955173287e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5\" (UID: \"b4adc0d7-6356-4632-8a7e-f0955173287e\") " pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"
Sep 09 20:30:43.939912 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.939809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgh4k\" (UniqueName: \"kubernetes.io/projected/b4adc0d7-6356-4632-8a7e-f0955173287e-kube-api-access-zgh4k\") pod \"payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5\" (UID: \"b4adc0d7-6356-4632-8a7e-f0955173287e\") " pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"
Sep 09 20:30:43.939912 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.939860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4adc0d7-6356-4632-8a7e-f0955173287e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5\" (UID: \"b4adc0d7-6356-4632-8a7e-f0955173287e\") " pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"
Sep 09 20:30:43.940696 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.940671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4adc0d7-6356-4632-8a7e-f0955173287e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5\" (UID: \"b4adc0d7-6356-4632-8a7e-f0955173287e\") " pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"
Sep 09 20:30:43.947266 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.947236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgh4k\" (UniqueName: \"kubernetes.io/projected/b4adc0d7-6356-4632-8a7e-f0955173287e-kube-api-access-zgh4k\") pod \"payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5\" (UID: \"b4adc0d7-6356-4632-8a7e-f0955173287e\") " pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"
Sep 09 20:30:43.994422 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:43.994382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:44.051985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:44.051947    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"
Sep 09 20:30:44.250627 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:44.250590    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:44.251018 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:44.250859    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:44.251636 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:44.251615    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:30:44.331210 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:44.331106    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"]
Sep 09 20:30:44.333782 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:30:44.333752    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4adc0d7_6356_4632_8a7e_f0955173287e.slice/crio-1bc659a02dde764373701253e6bf00c7d436b5a709a22207d1350654a5b7bf54 WatchSource:0}: Error finding container 1bc659a02dde764373701253e6bf00c7d436b5a709a22207d1350654a5b7bf54: Status 404 returned error can't find the container with id 1bc659a02dde764373701253e6bf00c7d436b5a709a22207d1350654a5b7bf54
Sep 09 20:30:44.797093 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:44.797054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:44.829679 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:44.829639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5" event={"ID":"b4adc0d7-6356-4632-8a7e-f0955173287e","Type":"ContainerStarted","Data":"c4ccdcf23145fd48485ec52d0f0215386d56b195a3c150bec0b63d250bb5942b"}
Sep 09 20:30:44.829860 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:44.829694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5" event={"ID":"b4adc0d7-6356-4632-8a7e-f0955173287e","Type":"ContainerStarted","Data":"1bc659a02dde764373701253e6bf00c7d436b5a709a22207d1350654a5b7bf54"}
Sep 09 20:30:44.829860 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:44.829746    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"
Sep 09 20:30:44.845224 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:44.845177    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5" podStartSLOduration=1.8451618760000001 podStartE2EDuration="1.845161876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:44.84427206 +0000 UTC m=+852.083374361" watchObservedRunningTime="2026-09-09 20:30:44.845161876 +0000 UTC m=+852.084264176"
Sep 09 20:30:51.542264 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:51.542220    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:51.542264 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:51.542270    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:51.555025 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:51.554995    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:51.880601 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:51.880510    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:30:55.836764 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:55.836731    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"
Sep 09 20:30:56.912772 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:30:56.912728    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:31:05.168534 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:05.168497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:05.992156 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:05.992118    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:06.785787 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:06.785747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:07.428623 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:07.428591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:08.236227 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:08.236187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:09.075328 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:09.075286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:11.556880 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:11.556816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:11.607502 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:11.607472    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:31:11.845825 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:11.845735    2563 patch_prober.go:28] interesting pod/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 20:31:11.845998 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:11.845833    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5" podUID="4cbb079f-0902-45d2-8d3d-6faee0a084e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:31:12.070275 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.070227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:12.151815 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.151728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:12.175634 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.175599    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"]
Sep 09 20:31:12.175926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.175871    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5" podUID="b4adc0d7-6356-4632-8a7e-f0955173287e" containerName="payload-processing" containerID="cri-o://c4ccdcf23145fd48485ec52d0f0215386d56b195a3c150bec0b63d250bb5942b" gracePeriod=30
Sep 09 20:31:12.218355 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.218317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:12.317383 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.317318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:12.372755 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.372723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:12.432107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.432078    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"
Sep 09 20:31:12.486334 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.486298    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"]
Sep 09 20:31:12.486603 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.486514    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5" podUID="4cbb079f-0902-45d2-8d3d-6faee0a084e4" containerName="istio-proxy" containerID="cri-o://9df18472e6c3b42c6154a562eb5fb778cd210a757b4bc780c79a9af85d8ce566" gracePeriod=30
Sep 09 20:31:12.514894 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.514855    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4adc0d7-6356-4632-8a7e-f0955173287e-plugins-config-volume\") pod \"b4adc0d7-6356-4632-8a7e-f0955173287e\" (UID: \"b4adc0d7-6356-4632-8a7e-f0955173287e\") "
Sep 09 20:31:12.515079 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.515000    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgh4k\" (UniqueName: \"kubernetes.io/projected/b4adc0d7-6356-4632-8a7e-f0955173287e-kube-api-access-zgh4k\") pod \"b4adc0d7-6356-4632-8a7e-f0955173287e\" (UID: \"b4adc0d7-6356-4632-8a7e-f0955173287e\") "
Sep 09 20:31:12.515342 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.515308    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4adc0d7-6356-4632-8a7e-f0955173287e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4adc0d7-6356-4632-8a7e-f0955173287e" (UID: "b4adc0d7-6356-4632-8a7e-f0955173287e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:12.517130 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.517107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4adc0d7-6356-4632-8a7e-f0955173287e-kube-api-access-zgh4k" (OuterVolumeSpecName: "kube-api-access-zgh4k") pod "b4adc0d7-6356-4632-8a7e-f0955173287e" (UID: "b4adc0d7-6356-4632-8a7e-f0955173287e"). InnerVolumeSpecName "kube-api-access-zgh4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:12.524781 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.524758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:12.616778 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.616736    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4adc0d7-6356-4632-8a7e-f0955173287e-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:12.616778 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.616780    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgh4k\" (UniqueName: \"kubernetes.io/projected/b4adc0d7-6356-4632-8a7e-f0955173287e-kube-api-access-zgh4k\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:12.794357 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.794307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:12.973645 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.973610    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4adc0d7-6356-4632-8a7e-f0955173287e" containerID="c4ccdcf23145fd48485ec52d0f0215386d56b195a3c150bec0b63d250bb5942b" exitCode=0
Sep 09 20:31:12.973815 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.973665    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"
Sep 09 20:31:12.973815 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.973688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5" event={"ID":"b4adc0d7-6356-4632-8a7e-f0955173287e","Type":"ContainerDied","Data":"c4ccdcf23145fd48485ec52d0f0215386d56b195a3c150bec0b63d250bb5942b"}
Sep 09 20:31:12.973815 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.973731    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5" event={"ID":"b4adc0d7-6356-4632-8a7e-f0955173287e","Type":"ContainerDied","Data":"1bc659a02dde764373701253e6bf00c7d436b5a709a22207d1350654a5b7bf54"}
Sep 09 20:31:12.973815 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.973751    2563 scope.go:117] "RemoveContainer" containerID="c4ccdcf23145fd48485ec52d0f0215386d56b195a3c150bec0b63d250bb5942b"
Sep 09 20:31:12.982809 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.982792    2563 scope.go:117] "RemoveContainer" containerID="c4ccdcf23145fd48485ec52d0f0215386d56b195a3c150bec0b63d250bb5942b"
Sep 09 20:31:12.983116 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:31:12.983090    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ccdcf23145fd48485ec52d0f0215386d56b195a3c150bec0b63d250bb5942b\": container with ID starting with c4ccdcf23145fd48485ec52d0f0215386d56b195a3c150bec0b63d250bb5942b not found: ID does not exist" containerID="c4ccdcf23145fd48485ec52d0f0215386d56b195a3c150bec0b63d250bb5942b"
Sep 09 20:31:12.983206 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.983129    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ccdcf23145fd48485ec52d0f0215386d56b195a3c150bec0b63d250bb5942b"} err="failed to get container status \"c4ccdcf23145fd48485ec52d0f0215386d56b195a3c150bec0b63d250bb5942b\": rpc error: code = NotFound desc = could not find container \"c4ccdcf23145fd48485ec52d0f0215386d56b195a3c150bec0b63d250bb5942b\": container with ID starting with c4ccdcf23145fd48485ec52d0f0215386d56b195a3c150bec0b63d250bb5942b not found: ID does not exist"
Sep 09 20:31:12.999413 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:12.999385    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"]
Sep 09 20:31:13.002749 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.002728    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-25e8b2a0-6db7c58d94-slqk5"]
Sep 09 20:31:13.032412 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.032372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:13.313242 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.313212    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4adc0d7-6356-4632-8a7e-f0955173287e" path="/var/lib/kubelet/pods/b4adc0d7-6356-4632-8a7e-f0955173287e/volumes"
Sep 09 20:31:13.428449 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.428409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:13.558835 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.558799    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:31:13.606016 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.605981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:13.625172 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625145    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-podinfo\") pod \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") "
Sep 09 20:31:13.625536 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-envoy\") pod \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") "
Sep 09 20:31:13.625536 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625221    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-workload-socket\") pod \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") "
Sep 09 20:31:13.625536 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625335    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-credential-socket\") pod \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") "
Sep 09 20:31:13.625536 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625387    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-data\") pod \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") "
Sep 09 20:31:13.625536 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625419    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-token\") pod \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") "
Sep 09 20:31:13.625536 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625459    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-workload-certs\") pod \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") "
Sep 09 20:31:13.625536 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625485    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4cbb079f-0902-45d2-8d3d-6faee0a084e4" (UID: "4cbb079f-0902-45d2-8d3d-6faee0a084e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:31:13.625536 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625489    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnp55\" (UniqueName: \"kubernetes.io/projected/4cbb079f-0902-45d2-8d3d-6faee0a084e4-kube-api-access-tnp55\") pod \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") "
Sep 09 20:31:13.625944 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625567    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4cbb079f-0902-45d2-8d3d-6faee0a084e4" (UID: "4cbb079f-0902-45d2-8d3d-6faee0a084e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:31:13.625944 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625608    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "4cbb079f-0902-45d2-8d3d-6faee0a084e4" (UID: "4cbb079f-0902-45d2-8d3d-6faee0a084e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:31:13.625944 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625622    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istiod-ca-cert\") pod \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\" (UID: \"4cbb079f-0902-45d2-8d3d-6faee0a084e4\") "
Sep 09 20:31:13.625944 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625755    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4cbb079f-0902-45d2-8d3d-6faee0a084e4" (UID: "4cbb079f-0902-45d2-8d3d-6faee0a084e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:31:13.626094 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625959    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:13.626094 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625956    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4cbb079f-0902-45d2-8d3d-6faee0a084e4" (UID: "4cbb079f-0902-45d2-8d3d-6faee0a084e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:13.626094 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.625979    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:13.626094 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.626009    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:13.626094 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.626026    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:13.627551 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.627519    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbb079f-0902-45d2-8d3d-6faee0a084e4-kube-api-access-tnp55" (OuterVolumeSpecName: "kube-api-access-tnp55") pod "4cbb079f-0902-45d2-8d3d-6faee0a084e4" (UID: "4cbb079f-0902-45d2-8d3d-6faee0a084e4"). InnerVolumeSpecName "kube-api-access-tnp55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:13.627673 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.627556    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "4cbb079f-0902-45d2-8d3d-6faee0a084e4" (UID: "4cbb079f-0902-45d2-8d3d-6faee0a084e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:13.627823 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.627797    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4cbb079f-0902-45d2-8d3d-6faee0a084e4" (UID: "4cbb079f-0902-45d2-8d3d-6faee0a084e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:31:13.627978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.627964    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4cbb079f-0902-45d2-8d3d-6faee0a084e4" (UID: "4cbb079f-0902-45d2-8d3d-6faee0a084e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:31:13.706257 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.706223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:13.726995 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.726963    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:13.726995 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.726989    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnp55\" (UniqueName: \"kubernetes.io/projected/4cbb079f-0902-45d2-8d3d-6faee0a084e4-kube-api-access-tnp55\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:13.726995 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.726999    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:13.727206 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.727008    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:13.727206 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.727019    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cbb079f-0902-45d2-8d3d-6faee0a084e4-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:13.980186 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.980155    2563 generic.go:356] "Generic (PLEG): container finished" podID="4cbb079f-0902-45d2-8d3d-6faee0a084e4" containerID="9df18472e6c3b42c6154a562eb5fb778cd210a757b4bc780c79a9af85d8ce566" exitCode=0
Sep 09 20:31:13.980367 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.980205    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"
Sep 09 20:31:13.980367 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.980243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5" event={"ID":"4cbb079f-0902-45d2-8d3d-6faee0a084e4","Type":"ContainerDied","Data":"9df18472e6c3b42c6154a562eb5fb778cd210a757b4bc780c79a9af85d8ce566"}
Sep 09 20:31:13.980367 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.980300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5" event={"ID":"4cbb079f-0902-45d2-8d3d-6faee0a084e4","Type":"ContainerDied","Data":"1dde67a7b3eb63f88815646d3576ce64771a4e677cb8c5601457b1746c2f42f6"}
Sep 09 20:31:13.980367 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.980329    2563 scope.go:117] "RemoveContainer" containerID="9df18472e6c3b42c6154a562eb5fb778cd210a757b4bc780c79a9af85d8ce566"
Sep 09 20:31:13.991485 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.991449    2563 scope.go:117] "RemoveContainer" containerID="9df18472e6c3b42c6154a562eb5fb778cd210a757b4bc780c79a9af85d8ce566"
Sep 09 20:31:13.992281 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:31:13.992254    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df18472e6c3b42c6154a562eb5fb778cd210a757b4bc780c79a9af85d8ce566\": container with ID starting with 9df18472e6c3b42c6154a562eb5fb778cd210a757b4bc780c79a9af85d8ce566 not found: ID does not exist" containerID="9df18472e6c3b42c6154a562eb5fb778cd210a757b4bc780c79a9af85d8ce566"
Sep 09 20:31:13.992402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:13.992286    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df18472e6c3b42c6154a562eb5fb778cd210a757b4bc780c79a9af85d8ce566"} err="failed to get container status \"9df18472e6c3b42c6154a562eb5fb778cd210a757b4bc780c79a9af85d8ce566\": rpc error: code = NotFound desc = could not find container \"9df18472e6c3b42c6154a562eb5fb778cd210a757b4bc780c79a9af85d8ce566\": container with ID starting with 9df18472e6c3b42c6154a562eb5fb778cd210a757b4bc780c79a9af85d8ce566 not found: ID does not exist"
Sep 09 20:31:14.008465 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:14.008440    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"]
Sep 09 20:31:14.013514 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:14.013488    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-25e8b2a0-gw-openshift-default-778b98bd8-ltff5"]
Sep 09 20:31:14.478332 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:14.478293    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:15.313817 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:15.313786    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cbb079f-0902-45d2-8d3d-6faee0a084e4" path="/var/lib/kubelet/pods/4cbb079f-0902-45d2-8d3d-6faee0a084e4/volumes"
Sep 09 20:31:15.372595 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:15.372540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:16.173347 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:16.173311    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:16.296223 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:16.296184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:16.396330 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:16.396288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:17.001611 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:17.001561    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:17.901727 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:17.901690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:18.783605 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:18.783545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:19.555883 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:19.555844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:19.656488 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:19.656453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:19.966281 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:19.966239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:20.064307 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.064270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:20.107313 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.107277    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"]
Sep 09 20:31:20.107639 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.107627    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbb079f-0902-45d2-8d3d-6faee0a084e4" containerName="istio-proxy"
Sep 09 20:31:20.107690 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.107641    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbb079f-0902-45d2-8d3d-6faee0a084e4" containerName="istio-proxy"
Sep 09 20:31:20.107690 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.107648    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4adc0d7-6356-4632-8a7e-f0955173287e" containerName="payload-processing"
Sep 09 20:31:20.107690 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.107653    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4adc0d7-6356-4632-8a7e-f0955173287e" containerName="payload-processing"
Sep 09 20:31:20.107791 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.107718    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cbb079f-0902-45d2-8d3d-6faee0a084e4" containerName="istio-proxy"
Sep 09 20:31:20.107791 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.107728    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4adc0d7-6356-4632-8a7e-f0955173287e" containerName="payload-processing"
Sep 09 20:31:20.110500 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.110481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"
Sep 09 20:31:20.113965 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.113942    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4275e4e8-dockercfg-9vxcf\""
Sep 09 20:31:20.114077 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.114008    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4275e4e8\""
Sep 09 20:31:20.125981 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.125960    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"]
Sep 09 20:31:20.167437 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.167404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:20.186349 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.186320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8846745-75bf-4b80-be44-2728034683dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm\" (UID: \"c8846745-75bf-4b80-be44-2728034683dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"
Sep 09 20:31:20.186459 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.186371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffcvc\" (UniqueName: \"kubernetes.io/projected/c8846745-75bf-4b80-be44-2728034683dc-kube-api-access-ffcvc\") pod \"payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm\" (UID: \"c8846745-75bf-4b80-be44-2728034683dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"
Sep 09 20:31:20.287681 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.287590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8846745-75bf-4b80-be44-2728034683dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm\" (UID: \"c8846745-75bf-4b80-be44-2728034683dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"
Sep 09 20:31:20.287681 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.287640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffcvc\" (UniqueName: \"kubernetes.io/projected/c8846745-75bf-4b80-be44-2728034683dc-kube-api-access-ffcvc\") pod \"payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm\" (UID: \"c8846745-75bf-4b80-be44-2728034683dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"
Sep 09 20:31:20.288344 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.288324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8846745-75bf-4b80-be44-2728034683dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm\" (UID: \"c8846745-75bf-4b80-be44-2728034683dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"
Sep 09 20:31:20.295234 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.295205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffcvc\" (UniqueName: \"kubernetes.io/projected/c8846745-75bf-4b80-be44-2728034683dc-kube-api-access-ffcvc\") pod \"payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm\" (UID: \"c8846745-75bf-4b80-be44-2728034683dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"
Sep 09 20:31:20.444047 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.443992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"
Sep 09 20:31:20.722849 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:20.722824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"]
Sep 09 20:31:20.725041 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:31:20.725013    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8846745_75bf_4b80_be44_2728034683dc.slice/crio-dae934d42db6b4ee87e0405e4879905e1357dd0c8fe136c63db89f2c6215129d WatchSource:0}: Error finding container dae934d42db6b4ee87e0405e4879905e1357dd0c8fe136c63db89f2c6215129d: Status 404 returned error can't find the container with id dae934d42db6b4ee87e0405e4879905e1357dd0c8fe136c63db89f2c6215129d
Sep 09 20:31:21.017019 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:21.016931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm" event={"ID":"c8846745-75bf-4b80-be44-2728034683dc","Type":"ContainerStarted","Data":"63f44d19ddad00c191a484b90c26b651f72e10971855e8513088b5095878a032"}
Sep 09 20:31:21.017019 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:21.016988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm" event={"ID":"c8846745-75bf-4b80-be44-2728034683dc","Type":"ContainerStarted","Data":"dae934d42db6b4ee87e0405e4879905e1357dd0c8fe136c63db89f2c6215129d"}
Sep 09 20:31:21.017219 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:21.017054    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"
Sep 09 20:31:21.032662 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:21.032610    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm" podStartSLOduration=1.032589617 podStartE2EDuration="1.032589617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:21.032008513 +0000 UTC m=+888.271110813" watchObservedRunningTime="2026-09-09 20:31:21.032589617 +0000 UTC m=+888.271691909"
Sep 09 20:31:31.706051 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:31.705964    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:31:32.024456 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:32.024370    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"
Sep 09 20:31:42.495982 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:42.495944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:43.310681 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:43.310646    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:44.137688 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:44.137656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:44.858755 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:44.858716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:45.809899 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:45.809860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:47.348510 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:47.348469    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j"]
Sep 09 20:31:47.352030 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:47.352011    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j"
Sep 09 20:31:47.354615 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:47.354589    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:31:47.354716 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:47.354638    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:31:47.358798 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:47.358776    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j"]
Sep 09 20:31:47.441453 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:47.441416    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12c6cc4-8f2f-4f3c-b4fe-899adb28f274-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j\" (UID: \"a12c6cc4-8f2f-4f3c-b4fe-899adb28f274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j"
Sep 09 20:31:47.542191 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:47.542151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12c6cc4-8f2f-4f3c-b4fe-899adb28f274-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j\" (UID: \"a12c6cc4-8f2f-4f3c-b4fe-899adb28f274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j"
Sep 09 20:31:47.542819 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:47.542800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12c6cc4-8f2f-4f3c-b4fe-899adb28f274-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j\" (UID: \"a12c6cc4-8f2f-4f3c-b4fe-899adb28f274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j"
Sep 09 20:31:47.664374 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:47.664283    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j"
Sep 09 20:31:47.902465 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:47.902430    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:31:47.931024 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:47.930954    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j"]
Sep 09 20:31:47.933954 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:31:47.933926    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12c6cc4_8f2f_4f3c_b4fe_899adb28f274.slice/crio-f48efd7fc0227a70de42c390814a259f857ecbe6ba2cb06feee1ba3b6078ede2 WatchSource:0}: Error finding container f48efd7fc0227a70de42c390814a259f857ecbe6ba2cb06feee1ba3b6078ede2: Status 404 returned error can't find the container with id f48efd7fc0227a70de42c390814a259f857ecbe6ba2cb06feee1ba3b6078ede2
Sep 09 20:31:48.144857 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:48.144823    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j" event={"ID":"a12c6cc4-8f2f-4f3c-b4fe-899adb28f274","Type":"ContainerStarted","Data":"f48efd7fc0227a70de42c390814a259f857ecbe6ba2cb06feee1ba3b6078ede2"}
Sep 09 20:31:48.290041 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:48.290006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:49.145876 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:49.145784    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:49.150919 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:49.150892    2563 generic.go:356] "Generic (PLEG): container finished" podID="a12c6cc4-8f2f-4f3c-b4fe-899adb28f274" containerID="9c2759074a4a0ea73dac9266b5235b3d1648e4c24948830c22c0fba7270562a0" exitCode=0
Sep 09 20:31:49.151044 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:49.150949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j" event={"ID":"a12c6cc4-8f2f-4f3c-b4fe-899adb28f274","Type":"ContainerDied","Data":"9c2759074a4a0ea73dac9266b5235b3d1648e4c24948830c22c0fba7270562a0"}
Sep 09 20:31:49.952077 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:49.952041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:50.281715 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:50.281691    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j"
Sep 09 20:31:50.472909 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:50.472815    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12c6cc4-8f2f-4f3c-b4fe-899adb28f274-maas-api-service-ca\") pod \"a12c6cc4-8f2f-4f3c-b4fe-899adb28f274\" (UID: \"a12c6cc4-8f2f-4f3c-b4fe-899adb28f274\") "
Sep 09 20:31:50.473187 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:50.473163    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12c6cc4-8f2f-4f3c-b4fe-899adb28f274-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a12c6cc4-8f2f-4f3c-b4fe-899adb28f274" (UID: "a12c6cc4-8f2f-4f3c-b4fe-899adb28f274"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:50.574553 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:50.574517    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12c6cc4-8f2f-4f3c-b4fe-899adb28f274-maas-api-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:50.610137 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:50.610093    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:51.162650 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:51.162617    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j"
Sep 09 20:31:51.162832 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:51.162656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j" event={"ID":"a12c6cc4-8f2f-4f3c-b4fe-899adb28f274","Type":"ContainerDied","Data":"f48efd7fc0227a70de42c390814a259f857ecbe6ba2cb06feee1ba3b6078ede2"}
Sep 09 20:31:51.162832 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:51.162732    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f48efd7fc0227a70de42c390814a259f857ecbe6ba2cb06feee1ba3b6078ede2"
Sep 09 20:31:51.310922 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:51.310885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:52.806730 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:52.806694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:57.127142 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:57.127097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:57.230143 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:57.230095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:58.112464 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.112422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:58.172222 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.172187    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"]
Sep 09 20:31:58.173126 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.173081    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk" podUID="03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b" containerName="payload-pre-processing" containerID="cri-o://481ca5107d3197f2e13e155c9e92236e6c9326823b9379903d4a96151d4b13fa" gracePeriod=30
Sep 09 20:31:58.206219 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.206186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:58.306183 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.306148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:58.386951 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.385940    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"]
Sep 09 20:31:58.386951 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.386329    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm" podUID="c8846745-75bf-4b80-be44-2728034683dc" containerName="payload-pre-processing" containerID="cri-o://63f44d19ddad00c191a484b90c26b651f72e10971855e8513088b5095878a032" gracePeriod=30
Sep 09 20:31:58.393549 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.393248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:58.485353 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.485320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:58.520716 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.520691    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"
Sep 09 20:31:58.542319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.542287    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b-plugins-config-volume\") pod \"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b\" (UID: \"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b\") "
Sep 09 20:31:58.542319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.542327    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s56kd\" (UniqueName: \"kubernetes.io/projected/03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b-kube-api-access-s56kd\") pod \"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b\" (UID: \"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b\") "
Sep 09 20:31:58.542761 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.542733    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b" (UID: "03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:58.545250 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.545217    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b-kube-api-access-s56kd" (OuterVolumeSpecName: "kube-api-access-s56kd") pod "03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b" (UID: "03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b"). InnerVolumeSpecName "kube-api-access-s56kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:58.642965 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.642658    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"
Sep 09 20:31:58.643152 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.643120    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:58.643227 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.643156    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s56kd\" (UniqueName: \"kubernetes.io/projected/03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b-kube-api-access-s56kd\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:58.744141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.744092    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffcvc\" (UniqueName: \"kubernetes.io/projected/c8846745-75bf-4b80-be44-2728034683dc-kube-api-access-ffcvc\") pod \"c8846745-75bf-4b80-be44-2728034683dc\" (UID: \"c8846745-75bf-4b80-be44-2728034683dc\") "
Sep 09 20:31:58.744349 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.744167    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8846745-75bf-4b80-be44-2728034683dc-plugins-config-volume\") pod \"c8846745-75bf-4b80-be44-2728034683dc\" (UID: \"c8846745-75bf-4b80-be44-2728034683dc\") "
Sep 09 20:31:58.744890 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.744854    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8846745-75bf-4b80-be44-2728034683dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c8846745-75bf-4b80-be44-2728034683dc" (UID: "c8846745-75bf-4b80-be44-2728034683dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:58.747120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.747091    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8846745-75bf-4b80-be44-2728034683dc-kube-api-access-ffcvc" (OuterVolumeSpecName: "kube-api-access-ffcvc") pod "c8846745-75bf-4b80-be44-2728034683dc" (UID: "c8846745-75bf-4b80-be44-2728034683dc"). InnerVolumeSpecName "kube-api-access-ffcvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:58.845460 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.845419    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffcvc\" (UniqueName: \"kubernetes.io/projected/c8846745-75bf-4b80-be44-2728034683dc-kube-api-access-ffcvc\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:58.845460 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.845458    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8846745-75bf-4b80-be44-2728034683dc-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:31:58.849226 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.849195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:58.944616 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.944495    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j"]
Sep 09 20:31:58.955463 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.955432    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4275e4e8-mj69j"]
Sep 09 20:31:58.976179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:58.976149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:59.074622 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.074586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:59.213654 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.213621    2563 generic.go:356] "Generic (PLEG): container finished" podID="03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b" containerID="481ca5107d3197f2e13e155c9e92236e6c9326823b9379903d4a96151d4b13fa" exitCode=0
Sep 09 20:31:59.214093 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.213667    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"
Sep 09 20:31:59.214093 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.213698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk" event={"ID":"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b","Type":"ContainerDied","Data":"481ca5107d3197f2e13e155c9e92236e6c9326823b9379903d4a96151d4b13fa"}
Sep 09 20:31:59.214093 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.213760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk" event={"ID":"03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b","Type":"ContainerDied","Data":"16911e98b2f58dcd19d69c75cc5e0f617ff7b1c48abffabf0fe2eff17dabe8d1"}
Sep 09 20:31:59.214093 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.213789    2563 scope.go:117] "RemoveContainer" containerID="481ca5107d3197f2e13e155c9e92236e6c9326823b9379903d4a96151d4b13fa"
Sep 09 20:31:59.215122 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.215070    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8846745-75bf-4b80-be44-2728034683dc" containerID="63f44d19ddad00c191a484b90c26b651f72e10971855e8513088b5095878a032" exitCode=0
Sep 09 20:31:59.215122 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.215100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm" event={"ID":"c8846745-75bf-4b80-be44-2728034683dc","Type":"ContainerDied","Data":"63f44d19ddad00c191a484b90c26b651f72e10971855e8513088b5095878a032"}
Sep 09 20:31:59.215122 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.215115    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"
Sep 09 20:31:59.215302 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.215154    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm" event={"ID":"c8846745-75bf-4b80-be44-2728034683dc","Type":"ContainerDied","Data":"dae934d42db6b4ee87e0405e4879905e1357dd0c8fe136c63db89f2c6215129d"}
Sep 09 20:31:59.222885 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.222870    2563 scope.go:117] "RemoveContainer" containerID="481ca5107d3197f2e13e155c9e92236e6c9326823b9379903d4a96151d4b13fa"
Sep 09 20:31:59.223211 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:31:59.223185    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481ca5107d3197f2e13e155c9e92236e6c9326823b9379903d4a96151d4b13fa\": container with ID starting with 481ca5107d3197f2e13e155c9e92236e6c9326823b9379903d4a96151d4b13fa not found: ID does not exist" containerID="481ca5107d3197f2e13e155c9e92236e6c9326823b9379903d4a96151d4b13fa"
Sep 09 20:31:59.223706 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.223219    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481ca5107d3197f2e13e155c9e92236e6c9326823b9379903d4a96151d4b13fa"} err="failed to get container status \"481ca5107d3197f2e13e155c9e92236e6c9326823b9379903d4a96151d4b13fa\": rpc error: code = NotFound desc = could not find container \"481ca5107d3197f2e13e155c9e92236e6c9326823b9379903d4a96151d4b13fa\": container with ID starting with 481ca5107d3197f2e13e155c9e92236e6c9326823b9379903d4a96151d4b13fa not found: ID does not exist"
Sep 09 20:31:59.223706 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.223625    2563 scope.go:117] "RemoveContainer" containerID="63f44d19ddad00c191a484b90c26b651f72e10971855e8513088b5095878a032"
Sep 09 20:31:59.237556 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.237531    2563 scope.go:117] "RemoveContainer" containerID="63f44d19ddad00c191a484b90c26b651f72e10971855e8513088b5095878a032"
Sep 09 20:31:59.237878 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:31:59.237858    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f44d19ddad00c191a484b90c26b651f72e10971855e8513088b5095878a032\": container with ID starting with 63f44d19ddad00c191a484b90c26b651f72e10971855e8513088b5095878a032 not found: ID does not exist" containerID="63f44d19ddad00c191a484b90c26b651f72e10971855e8513088b5095878a032"
Sep 09 20:31:59.237936 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.237887    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f44d19ddad00c191a484b90c26b651f72e10971855e8513088b5095878a032"} err="failed to get container status \"63f44d19ddad00c191a484b90c26b651f72e10971855e8513088b5095878a032\": rpc error: code = NotFound desc = could not find container \"63f44d19ddad00c191a484b90c26b651f72e10971855e8513088b5095878a032\": container with ID starting with 63f44d19ddad00c191a484b90c26b651f72e10971855e8513088b5095878a032 not found: ID does not exist"
Sep 09 20:31:59.254244 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.254205    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"]
Sep 09 20:31:59.264892 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.264859    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-812760-54fdb888f8-psnfk"]
Sep 09 20:31:59.276836 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.276806    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"]
Sep 09 20:31:59.281582 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.281540    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4275e4e8-f89b996b-rhbrm"]
Sep 09 20:31:59.313695 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.313654    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b" path="/var/lib/kubelet/pods/03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b/volumes"
Sep 09 20:31:59.314098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.314074    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a12c6cc4-8f2f-4f3c-b4fe-899adb28f274" path="/var/lib/kubelet/pods/a12c6cc4-8f2f-4f3c-b4fe-899adb28f274/volumes"
Sep 09 20:31:59.314474 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.314455    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8846745-75bf-4b80-be44-2728034683dc" path="/var/lib/kubelet/pods/c8846745-75bf-4b80-be44-2728034683dc/volumes"
Sep 09 20:31:59.993811 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:31:59.993773    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:00.105217 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:00.105178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:00.539162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:00.539129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:00.853009 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:00.852922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:00.972868 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:00.972816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:01.164634 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:01.164514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:01.726513 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:01.726477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:03.283487 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:03.283457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:03.462377 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:03.462341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:03.553851 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:03.553755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:03.901540 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:03.901450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:04.117393 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:04.117357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:04.902335 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:04.902301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:04.958801 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:04.958764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:05.088264 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.088227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:05.397840 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.397801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:05.472786 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.472749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:05.491997 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.491964    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"]
Sep 09 20:32:05.492285 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.492274    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12c6cc4-8f2f-4f3c-b4fe-899adb28f274" containerName="revoke-keys"
Sep 09 20:32:05.492333 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.492287    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12c6cc4-8f2f-4f3c-b4fe-899adb28f274" containerName="revoke-keys"
Sep 09 20:32:05.492333 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.492296    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8846745-75bf-4b80-be44-2728034683dc" containerName="payload-pre-processing"
Sep 09 20:32:05.492333 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.492302    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8846745-75bf-4b80-be44-2728034683dc" containerName="payload-pre-processing"
Sep 09 20:32:05.492333 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.492316    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b" containerName="payload-pre-processing"
Sep 09 20:32:05.492333 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.492321    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b" containerName="payload-pre-processing"
Sep 09 20:32:05.492484 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.492380    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a12c6cc4-8f2f-4f3c-b4fe-899adb28f274" containerName="revoke-keys"
Sep 09 20:32:05.492484 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.492387    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b5e00e-f738-4b8b-9bc4-9d62cd37fe4b" containerName="payload-pre-processing"
Sep 09 20:32:05.492484 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.492394    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8846745-75bf-4b80-be44-2728034683dc" containerName="payload-pre-processing"
Sep 09 20:32:05.499723 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.499689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"
Sep 09 20:32:05.502512 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.502486    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f4c2073a-dockercfg-m4qhq\""
Sep 09 20:32:05.503390 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.503366    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f4c2073a\""
Sep 09 20:32:05.504552 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.504527    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"]
Sep 09 20:32:05.526134 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.526103    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"]
Sep 09 20:32:05.528466 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.528450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"
Sep 09 20:32:05.538554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.538528    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"]
Sep 09 20:32:05.608098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.608060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm\" (UID: \"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"
Sep 09 20:32:05.608290 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.608130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn5jp\" (UniqueName: \"kubernetes.io/projected/a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e-kube-api-access-jn5jp\") pod \"payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm\" (UID: \"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"
Sep 09 20:32:05.608290 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.608167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7613f0db-2111-45b4-814f-71c4ffd40bcf-plugins-config-volume\") pod \"payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp\" (UID: \"7613f0db-2111-45b4-814f-71c4ffd40bcf\") " pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"
Sep 09 20:32:05.608290 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.608212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rw28\" (UniqueName: \"kubernetes.io/projected/7613f0db-2111-45b4-814f-71c4ffd40bcf-kube-api-access-9rw28\") pod \"payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp\" (UID: \"7613f0db-2111-45b4-814f-71c4ffd40bcf\") " pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"
Sep 09 20:32:05.708949 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.708912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn5jp\" (UniqueName: \"kubernetes.io/projected/a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e-kube-api-access-jn5jp\") pod \"payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm\" (UID: \"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"
Sep 09 20:32:05.709115 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.708964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7613f0db-2111-45b4-814f-71c4ffd40bcf-plugins-config-volume\") pod \"payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp\" (UID: \"7613f0db-2111-45b4-814f-71c4ffd40bcf\") " pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"
Sep 09 20:32:05.709115 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.709002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rw28\" (UniqueName: \"kubernetes.io/projected/7613f0db-2111-45b4-814f-71c4ffd40bcf-kube-api-access-9rw28\") pod \"payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp\" (UID: \"7613f0db-2111-45b4-814f-71c4ffd40bcf\") " pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"
Sep 09 20:32:05.709115 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.709076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm\" (UID: \"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"
Sep 09 20:32:05.710005 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.709979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm\" (UID: \"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"
Sep 09 20:32:05.710070 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.709979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7613f0db-2111-45b4-814f-71c4ffd40bcf-plugins-config-volume\") pod \"payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp\" (UID: \"7613f0db-2111-45b4-814f-71c4ffd40bcf\") " pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"
Sep 09 20:32:05.716435 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.716404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn5jp\" (UniqueName: \"kubernetes.io/projected/a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e-kube-api-access-jn5jp\") pod \"payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm\" (UID: \"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"
Sep 09 20:32:05.716601 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.716564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rw28\" (UniqueName: \"kubernetes.io/projected/7613f0db-2111-45b4-814f-71c4ffd40bcf-kube-api-access-9rw28\") pod \"payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp\" (UID: \"7613f0db-2111-45b4-814f-71c4ffd40bcf\") " pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"
Sep 09 20:32:05.811695 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.811655    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"
Sep 09 20:32:05.839018 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:05.838989    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"
Sep 09 20:32:06.085350 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:06.085325    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"]
Sep 09 20:32:06.087806 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:32:06.087778    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4dfe886_e8cc_4d22_af9a_ff4dc75f6e4e.slice/crio-bc7396e613a8b32ecf420e6f3c738258576989b4baf37a4edaf9ed71014eab03 WatchSource:0}: Error finding container bc7396e613a8b32ecf420e6f3c738258576989b4baf37a4edaf9ed71014eab03: Status 404 returned error can't find the container with id bc7396e613a8b32ecf420e6f3c738258576989b4baf37a4edaf9ed71014eab03
Sep 09 20:32:06.118235 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:06.118210    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"]
Sep 09 20:32:06.120048 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:32:06.120025    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7613f0db_2111_45b4_814f_71c4ffd40bcf.slice/crio-87f7ac4e30f36a66cdb7975ac2b40fa24db46dc3b97bac6dfc8f38c05394937f WatchSource:0}: Error finding container 87f7ac4e30f36a66cdb7975ac2b40fa24db46dc3b97bac6dfc8f38c05394937f: Status 404 returned error can't find the container with id 87f7ac4e30f36a66cdb7975ac2b40fa24db46dc3b97bac6dfc8f38c05394937f
Sep 09 20:32:06.256560 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:06.256514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp" event={"ID":"7613f0db-2111-45b4-814f-71c4ffd40bcf","Type":"ContainerStarted","Data":"7ee26a3e8f11971b46d59ab1945489d30ca52cf7f8463d10abd978db253ec914"}
Sep 09 20:32:06.256753 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:06.256599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp" event={"ID":"7613f0db-2111-45b4-814f-71c4ffd40bcf","Type":"ContainerStarted","Data":"87f7ac4e30f36a66cdb7975ac2b40fa24db46dc3b97bac6dfc8f38c05394937f"}
Sep 09 20:32:06.256753 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:06.256724    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"
Sep 09 20:32:06.258534 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:06.258505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm" event={"ID":"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e","Type":"ContainerStarted","Data":"e8f0950b145a729f1feec2a40c6c062e9e773707b194bc12495c4bfba0a324cb"}
Sep 09 20:32:06.258659 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:06.258558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm" event={"ID":"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e","Type":"ContainerStarted","Data":"bc7396e613a8b32ecf420e6f3c738258576989b4baf37a4edaf9ed71014eab03"}
Sep 09 20:32:06.258659 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:06.258610    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"
Sep 09 20:32:06.275139 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:06.275067    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp" podStartSLOduration=1.275045446 podStartE2EDuration="1.275045446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:06.273816849 +0000 UTC m=+933.512919152" watchObservedRunningTime="2026-09-09 20:32:06.275045446 +0000 UTC m=+933.514147740"
Sep 09 20:32:06.290745 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:06.290690    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm" podStartSLOduration=1.290674629 podStartE2EDuration="1.290674629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:06.289154307 +0000 UTC m=+933.528256607" watchObservedRunningTime="2026-09-09 20:32:06.290674629 +0000 UTC m=+933.529776928"
Sep 09 20:32:07.001880 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:07.001845    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:32:15.178419 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:15.178377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:17.043192 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:17.043159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:17.266681 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:17.266643    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"
Sep 09 20:32:17.266853 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:17.266806    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"
Sep 09 20:32:22.611969 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:22.611931    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:32:24.053454 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:24.053412    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:26.758145 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:26.758098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:28.026750 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:28.026714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:28.349901 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:28.349816    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8"]
Sep 09 20:32:28.354838 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:28.354819    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8"
Sep 09 20:32:28.358278 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:28.358252    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:32:28.358631 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:28.358341    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:32:28.360710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:28.360690    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8"]
Sep 09 20:32:28.525698 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:28.525660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31df38-189a-45c1-8ada-fc80d8839f98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8\" (UID: \"ac31df38-189a-45c1-8ada-fc80d8839f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8"
Sep 09 20:32:28.626512 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:28.626423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31df38-189a-45c1-8ada-fc80d8839f98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8\" (UID: \"ac31df38-189a-45c1-8ada-fc80d8839f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8"
Sep 09 20:32:28.627099 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:28.627076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31df38-189a-45c1-8ada-fc80d8839f98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8\" (UID: \"ac31df38-189a-45c1-8ada-fc80d8839f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8"
Sep 09 20:32:28.667634 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:28.667597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8"
Sep 09 20:32:28.935983 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:28.935927    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8"]
Sep 09 20:32:28.938386 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:32:28.938357    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac31df38_189a_45c1_8ada_fc80d8839f98.slice/crio-2d6ecc140bdbfb2e595de3cccd95b1a70755edbe3b5f0f879aab22d2fa19ff03 WatchSource:0}: Error finding container 2d6ecc140bdbfb2e595de3cccd95b1a70755edbe3b5f0f879aab22d2fa19ff03: Status 404 returned error can't find the container with id 2d6ecc140bdbfb2e595de3cccd95b1a70755edbe3b5f0f879aab22d2fa19ff03
Sep 09 20:32:28.996015 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:28.995986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:29.372298 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:29.372259    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8" event={"ID":"ac31df38-189a-45c1-8ada-fc80d8839f98","Type":"ContainerStarted","Data":"8062778a5e7b2a8e0b6bad8416cb19ec31c88f0ffa39a45cbbe7058224d49b13"}
Sep 09 20:32:29.372712 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:29.372309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8" event={"ID":"ac31df38-189a-45c1-8ada-fc80d8839f98","Type":"ContainerStarted","Data":"2d6ecc140bdbfb2e595de3cccd95b1a70755edbe3b5f0f879aab22d2fa19ff03"}
Sep 09 20:32:29.389399 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:29.389351    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8" podStartSLOduration=1.389335685 podStartE2EDuration="1.389335685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:29.388303381 +0000 UTC m=+956.627405681" watchObservedRunningTime="2026-09-09 20:32:29.389335685 +0000 UTC m=+956.628437985"
Sep 09 20:32:29.563553 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:29.563467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:29.781320 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:29.781283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:30.378549 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:30.378514    2563 generic.go:356] "Generic (PLEG): container finished" podID="ac31df38-189a-45c1-8ada-fc80d8839f98" containerID="8062778a5e7b2a8e0b6bad8416cb19ec31c88f0ffa39a45cbbe7058224d49b13" exitCode=0
Sep 09 20:32:30.378950 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:30.378597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8" event={"ID":"ac31df38-189a-45c1-8ada-fc80d8839f98","Type":"ContainerDied","Data":"8062778a5e7b2a8e0b6bad8416cb19ec31c88f0ffa39a45cbbe7058224d49b13"}
Sep 09 20:32:30.692496 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:30.692403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:31.413991 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:31.413953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:31.519441 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:31.519411    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8"
Sep 09 20:32:31.657002 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:31.656912    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31df38-189a-45c1-8ada-fc80d8839f98-maas-api-service-ca\") pod \"ac31df38-189a-45c1-8ada-fc80d8839f98\" (UID: \"ac31df38-189a-45c1-8ada-fc80d8839f98\") "
Sep 09 20:32:31.657325 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:31.657294    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac31df38-189a-45c1-8ada-fc80d8839f98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac31df38-189a-45c1-8ada-fc80d8839f98" (UID: "ac31df38-189a-45c1-8ada-fc80d8839f98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:31.757937 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:31.757903    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31df38-189a-45c1-8ada-fc80d8839f98-maas-api-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:32.390092 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:32.390059    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8"
Sep 09 20:32:32.390278 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:32.390088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8" event={"ID":"ac31df38-189a-45c1-8ada-fc80d8839f98","Type":"ContainerDied","Data":"2d6ecc140bdbfb2e595de3cccd95b1a70755edbe3b5f0f879aab22d2fa19ff03"}
Sep 09 20:32:32.390278 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:32.390136    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d6ecc140bdbfb2e595de3cccd95b1a70755edbe3b5f0f879aab22d2fa19ff03"
Sep 09 20:32:34.876378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:34.876337    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:35.482933 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.482896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:35.495952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.495918    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"]
Sep 09 20:32:35.496154 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.496125    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp" podUID="7613f0db-2111-45b4-814f-71c4ffd40bcf" containerName="payload-processing" containerID="cri-o://7ee26a3e8f11971b46d59ab1945489d30ca52cf7f8463d10abd978db253ec914" gracePeriod=30
Sep 09 20:32:35.504503 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.504056    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"]
Sep 09 20:32:35.504503 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.504318    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm" podUID="a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e" containerName="payload-pre-processing" containerID="cri-o://e8f0950b145a729f1feec2a40c6c062e9e773707b194bc12495c4bfba0a324cb" gracePeriod=30
Sep 09 20:32:35.582780 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.582749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:35.670542 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.670491    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8"]
Sep 09 20:32:35.672669 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.672624    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f4c2073a-4gjt8"]
Sep 09 20:32:35.672669 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.672631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:35.752533 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.752499    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:35.757220 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.757195    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"
Sep 09 20:32:35.782369 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.782344    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"
Sep 09 20:32:35.824805 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.824773    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:35.895897 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.895859    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e-plugins-config-volume\") pod \"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e\" (UID: \"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e\") "
Sep 09 20:32:35.896368 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.895922    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn5jp\" (UniqueName: \"kubernetes.io/projected/a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e-kube-api-access-jn5jp\") pod \"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e\" (UID: \"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e\") "
Sep 09 20:32:35.896368 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.896016    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rw28\" (UniqueName: \"kubernetes.io/projected/7613f0db-2111-45b4-814f-71c4ffd40bcf-kube-api-access-9rw28\") pod \"7613f0db-2111-45b4-814f-71c4ffd40bcf\" (UID: \"7613f0db-2111-45b4-814f-71c4ffd40bcf\") "
Sep 09 20:32:35.896368 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.896046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7613f0db-2111-45b4-814f-71c4ffd40bcf-plugins-config-volume\") pod \"7613f0db-2111-45b4-814f-71c4ffd40bcf\" (UID: \"7613f0db-2111-45b4-814f-71c4ffd40bcf\") "
Sep 09 20:32:35.896368 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.896270    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e" (UID: "a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:35.901284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.896544    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7613f0db-2111-45b4-814f-71c4ffd40bcf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7613f0db-2111-45b4-814f-71c4ffd40bcf" (UID: "7613f0db-2111-45b4-814f-71c4ffd40bcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:35.901284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.899711    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e-kube-api-access-jn5jp" (OuterVolumeSpecName: "kube-api-access-jn5jp") pod "a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e" (UID: "a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e"). InnerVolumeSpecName "kube-api-access-jn5jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:35.901520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.901489    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7613f0db-2111-45b4-814f-71c4ffd40bcf-kube-api-access-9rw28" (OuterVolumeSpecName: "kube-api-access-9rw28") pod "7613f0db-2111-45b4-814f-71c4ffd40bcf" (UID: "7613f0db-2111-45b4-814f-71c4ffd40bcf"). InnerVolumeSpecName "kube-api-access-9rw28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:35.996968 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.996878    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:35.996968 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.996910    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn5jp\" (UniqueName: \"kubernetes.io/projected/a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e-kube-api-access-jn5jp\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:35.996968 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.996920    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rw28\" (UniqueName: \"kubernetes.io/projected/7613f0db-2111-45b4-814f-71c4ffd40bcf-kube-api-access-9rw28\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:35.996968 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:35.996929    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7613f0db-2111-45b4-814f-71c4ffd40bcf-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:36.086430 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.086392    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:36.409219 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.409134    2563 generic.go:356] "Generic (PLEG): container finished" podID="7613f0db-2111-45b4-814f-71c4ffd40bcf" containerID="7ee26a3e8f11971b46d59ab1945489d30ca52cf7f8463d10abd978db253ec914" exitCode=0
Sep 09 20:32:36.409219 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.409171    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"
Sep 09 20:32:36.409425 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.409215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp" event={"ID":"7613f0db-2111-45b4-814f-71c4ffd40bcf","Type":"ContainerDied","Data":"7ee26a3e8f11971b46d59ab1945489d30ca52cf7f8463d10abd978db253ec914"}
Sep 09 20:32:36.409425 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.409263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp" event={"ID":"7613f0db-2111-45b4-814f-71c4ffd40bcf","Type":"ContainerDied","Data":"87f7ac4e30f36a66cdb7975ac2b40fa24db46dc3b97bac6dfc8f38c05394937f"}
Sep 09 20:32:36.409425 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.409291    2563 scope.go:117] "RemoveContainer" containerID="7ee26a3e8f11971b46d59ab1945489d30ca52cf7f8463d10abd978db253ec914"
Sep 09 20:32:36.410473 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.410453    2563 generic.go:356] "Generic (PLEG): container finished" podID="a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e" containerID="e8f0950b145a729f1feec2a40c6c062e9e773707b194bc12495c4bfba0a324cb" exitCode=0
Sep 09 20:32:36.410612 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.410497    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"
Sep 09 20:32:36.410612 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.410551    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm" event={"ID":"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e","Type":"ContainerDied","Data":"e8f0950b145a729f1feec2a40c6c062e9e773707b194bc12495c4bfba0a324cb"}
Sep 09 20:32:36.410743 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.410617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm" event={"ID":"a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e","Type":"ContainerDied","Data":"bc7396e613a8b32ecf420e6f3c738258576989b4baf37a4edaf9ed71014eab03"}
Sep 09 20:32:36.418169 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.418152    2563 scope.go:117] "RemoveContainer" containerID="7ee26a3e8f11971b46d59ab1945489d30ca52cf7f8463d10abd978db253ec914"
Sep 09 20:32:36.418457 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:32:36.418435    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee26a3e8f11971b46d59ab1945489d30ca52cf7f8463d10abd978db253ec914\": container with ID starting with 7ee26a3e8f11971b46d59ab1945489d30ca52cf7f8463d10abd978db253ec914 not found: ID does not exist" containerID="7ee26a3e8f11971b46d59ab1945489d30ca52cf7f8463d10abd978db253ec914"
Sep 09 20:32:36.418527 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.418466    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee26a3e8f11971b46d59ab1945489d30ca52cf7f8463d10abd978db253ec914"} err="failed to get container status \"7ee26a3e8f11971b46d59ab1945489d30ca52cf7f8463d10abd978db253ec914\": rpc error: code = NotFound desc = could not find container \"7ee26a3e8f11971b46d59ab1945489d30ca52cf7f8463d10abd978db253ec914\": container with ID starting with 7ee26a3e8f11971b46d59ab1945489d30ca52cf7f8463d10abd978db253ec914 not found: ID does not exist"
Sep 09 20:32:36.418527 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.418483    2563 scope.go:117] "RemoveContainer" containerID="e8f0950b145a729f1feec2a40c6c062e9e773707b194bc12495c4bfba0a324cb"
Sep 09 20:32:36.426303 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.426287    2563 scope.go:117] "RemoveContainer" containerID="e8f0950b145a729f1feec2a40c6c062e9e773707b194bc12495c4bfba0a324cb"
Sep 09 20:32:36.427143 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:32:36.426564    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f0950b145a729f1feec2a40c6c062e9e773707b194bc12495c4bfba0a324cb\": container with ID starting with e8f0950b145a729f1feec2a40c6c062e9e773707b194bc12495c4bfba0a324cb not found: ID does not exist" containerID="e8f0950b145a729f1feec2a40c6c062e9e773707b194bc12495c4bfba0a324cb"
Sep 09 20:32:36.427200 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.427152    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f0950b145a729f1feec2a40c6c062e9e773707b194bc12495c4bfba0a324cb"} err="failed to get container status \"e8f0950b145a729f1feec2a40c6c062e9e773707b194bc12495c4bfba0a324cb\": rpc error: code = NotFound desc = could not find container \"e8f0950b145a729f1feec2a40c6c062e9e773707b194bc12495c4bfba0a324cb\": container with ID starting with e8f0950b145a729f1feec2a40c6c062e9e773707b194bc12495c4bfba0a324cb not found: ID does not exist"
Sep 09 20:32:36.434749 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.434728    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"]
Sep 09 20:32:36.438506 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.438486    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f4c2073a-584b7876f-7rgxp"]
Sep 09 20:32:36.448395 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.448370    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"]
Sep 09 20:32:36.453986 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.453965    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f4c2073a-86cbffcbc8-krgdm"]
Sep 09 20:32:36.519699 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.519668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:36.618203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.618165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:36.718864 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:36.718830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:37.063229 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:37.063138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:37.313607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:37.313511    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7613f0db-2111-45b4-814f-71c4ffd40bcf" path="/var/lib/kubelet/pods/7613f0db-2111-45b4-814f-71c4ffd40bcf/volumes"
Sep 09 20:32:37.313910 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:37.313895    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e" path="/var/lib/kubelet/pods/a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e/volumes"
Sep 09 20:32:37.314195 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:37.314184    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac31df38-189a-45c1-8ada-fc80d8839f98" path="/var/lib/kubelet/pods/ac31df38-189a-45c1-8ada-fc80d8839f98/volumes"
Sep 09 20:32:37.759766 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:37.759729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:38.190884 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:38.190798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:38.679402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:38.679363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:38.911930 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:38.911895    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"]
Sep 09 20:32:38.912238 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:38.912209    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" containerName="istio-proxy" containerID="cri-o://8c236f9e2dcda96c18a7c453b7836133cc6485e4a830d600c9fd202a0f483a50" gracePeriod=30
Sep 09 20:32:38.938534 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:38.938454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:39.446265 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:39.446225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:39.722483 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:39.722448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:40.314367 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:40.314330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:41.699698 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:41.699661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:32:43.081522 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:43.081484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:43.335213 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:43.335089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:43.486771 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:43.486736    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:44.034387 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.034357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:44.172515 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.172491    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:32:44.265465 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.265436    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxn2g\" (UniqueName: \"kubernetes.io/projected/69d67d76-d49b-451c-93ab-a9e53afda094-kube-api-access-wxn2g\") pod \"69d67d76-d49b-451c-93ab-a9e53afda094\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") "
Sep 09 20:32:44.265681 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.265484    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-credential-socket\") pod \"69d67d76-d49b-451c-93ab-a9e53afda094\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") "
Sep 09 20:32:44.265681 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.265556    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-workload-socket\") pod \"69d67d76-d49b-451c-93ab-a9e53afda094\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") "
Sep 09 20:32:44.265681 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.265611    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d67d76-d49b-451c-93ab-a9e53afda094-istiod-ca-cert\") pod \"69d67d76-d49b-451c-93ab-a9e53afda094\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") "
Sep 09 20:32:44.265681 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.265646    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-istio-envoy\") pod \"69d67d76-d49b-451c-93ab-a9e53afda094\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") "
Sep 09 20:32:44.265903 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.265682    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d67d76-d49b-451c-93ab-a9e53afda094-istio-token\") pod \"69d67d76-d49b-451c-93ab-a9e53afda094\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") "
Sep 09 20:32:44.265903 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.265704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-istio-data\") pod \"69d67d76-d49b-451c-93ab-a9e53afda094\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") "
Sep 09 20:32:44.266006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.265900    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "69d67d76-d49b-451c-93ab-a9e53afda094" (UID: "69d67d76-d49b-451c-93ab-a9e53afda094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:44.266006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.265910    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "69d67d76-d49b-451c-93ab-a9e53afda094" (UID: "69d67d76-d49b-451c-93ab-a9e53afda094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:44.266006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.265966    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-workload-certs\") pod \"69d67d76-d49b-451c-93ab-a9e53afda094\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") "
Sep 09 20:32:44.266006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.265996    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-istio-data" (OuterVolumeSpecName: "istio-data") pod "69d67d76-d49b-451c-93ab-a9e53afda094" (UID: "69d67d76-d49b-451c-93ab-a9e53afda094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:44.266211 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.266007    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d67d76-d49b-451c-93ab-a9e53afda094-istio-podinfo\") pod \"69d67d76-d49b-451c-93ab-a9e53afda094\" (UID: \"69d67d76-d49b-451c-93ab-a9e53afda094\") "
Sep 09 20:32:44.266211 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.266171    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "69d67d76-d49b-451c-93ab-a9e53afda094" (UID: "69d67d76-d49b-451c-93ab-a9e53afda094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:44.266211 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.266189    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d67d76-d49b-451c-93ab-a9e53afda094-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "69d67d76-d49b-451c-93ab-a9e53afda094" (UID: "69d67d76-d49b-451c-93ab-a9e53afda094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:44.266348 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.266336    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:44.266405 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.266355    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d67d76-d49b-451c-93ab-a9e53afda094-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:44.266405 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.266369    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:44.266405 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.266383    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:44.266405 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.266396    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:44.267889 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.267852    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "69d67d76-d49b-451c-93ab-a9e53afda094" (UID: "69d67d76-d49b-451c-93ab-a9e53afda094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:44.268240 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.268209    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69d67d76-d49b-451c-93ab-a9e53afda094-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "69d67d76-d49b-451c-93ab-a9e53afda094" (UID: "69d67d76-d49b-451c-93ab-a9e53afda094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:32:44.268332 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.268248    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d67d76-d49b-451c-93ab-a9e53afda094-kube-api-access-wxn2g" (OuterVolumeSpecName: "kube-api-access-wxn2g") pod "69d67d76-d49b-451c-93ab-a9e53afda094" (UID: "69d67d76-d49b-451c-93ab-a9e53afda094"). InnerVolumeSpecName "kube-api-access-wxn2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:44.268332 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.268260    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d67d76-d49b-451c-93ab-a9e53afda094-istio-token" (OuterVolumeSpecName: "istio-token") pod "69d67d76-d49b-451c-93ab-a9e53afda094" (UID: "69d67d76-d49b-451c-93ab-a9e53afda094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:44.367033 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.367000    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d67d76-d49b-451c-93ab-a9e53afda094-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:44.367033 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.367028    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d67d76-d49b-451c-93ab-a9e53afda094-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:44.367033 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.367038    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d67d76-d49b-451c-93ab-a9e53afda094-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:44.367033 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.367047    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxn2g\" (UniqueName: \"kubernetes.io/projected/69d67d76-d49b-451c-93ab-a9e53afda094-kube-api-access-wxn2g\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:32:44.453486 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.453451    2563 generic.go:356] "Generic (PLEG): container finished" podID="69d67d76-d49b-451c-93ab-a9e53afda094" containerID="8c236f9e2dcda96c18a7c453b7836133cc6485e4a830d600c9fd202a0f483a50" exitCode=0
Sep 09 20:32:44.453486 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.453489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" event={"ID":"69d67d76-d49b-451c-93ab-a9e53afda094","Type":"ContainerDied","Data":"8c236f9e2dcda96c18a7c453b7836133cc6485e4a830d600c9fd202a0f483a50"}
Sep 09 20:32:44.453781 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.453497    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"
Sep 09 20:32:44.453781 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.453528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh" event={"ID":"69d67d76-d49b-451c-93ab-a9e53afda094","Type":"ContainerDied","Data":"f5f30243ad27bed66d168d1358b573b84370718f8c40f709b1c85aaf1aa4a165"}
Sep 09 20:32:44.453781 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.453549    2563 scope.go:117] "RemoveContainer" containerID="8c236f9e2dcda96c18a7c453b7836133cc6485e4a830d600c9fd202a0f483a50"
Sep 09 20:32:44.462757 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.462737    2563 scope.go:117] "RemoveContainer" containerID="f7e299e6246c0c0adf5bd2ee7b4d21ac312720ef122c69639d42b42f70f50e30"
Sep 09 20:32:44.471638 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.471584    2563 scope.go:117] "RemoveContainer" containerID="8c236f9e2dcda96c18a7c453b7836133cc6485e4a830d600c9fd202a0f483a50"
Sep 09 20:32:44.472032 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:32:44.472007    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c236f9e2dcda96c18a7c453b7836133cc6485e4a830d600c9fd202a0f483a50\": container with ID starting with 8c236f9e2dcda96c18a7c453b7836133cc6485e4a830d600c9fd202a0f483a50 not found: ID does not exist" containerID="8c236f9e2dcda96c18a7c453b7836133cc6485e4a830d600c9fd202a0f483a50"
Sep 09 20:32:44.472149 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.472044    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c236f9e2dcda96c18a7c453b7836133cc6485e4a830d600c9fd202a0f483a50"} err="failed to get container status \"8c236f9e2dcda96c18a7c453b7836133cc6485e4a830d600c9fd202a0f483a50\": rpc error: code = NotFound desc = could not find container \"8c236f9e2dcda96c18a7c453b7836133cc6485e4a830d600c9fd202a0f483a50\": container with ID starting with 8c236f9e2dcda96c18a7c453b7836133cc6485e4a830d600c9fd202a0f483a50 not found: ID does not exist"
Sep 09 20:32:44.472149 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.472082    2563 scope.go:117] "RemoveContainer" containerID="f7e299e6246c0c0adf5bd2ee7b4d21ac312720ef122c69639d42b42f70f50e30"
Sep 09 20:32:44.472525 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:32:44.472497    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e299e6246c0c0adf5bd2ee7b4d21ac312720ef122c69639d42b42f70f50e30\": container with ID starting with f7e299e6246c0c0adf5bd2ee7b4d21ac312720ef122c69639d42b42f70f50e30 not found: ID does not exist" containerID="f7e299e6246c0c0adf5bd2ee7b4d21ac312720ef122c69639d42b42f70f50e30"
Sep 09 20:32:44.472631 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.472538    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e299e6246c0c0adf5bd2ee7b4d21ac312720ef122c69639d42b42f70f50e30"} err="failed to get container status \"f7e299e6246c0c0adf5bd2ee7b4d21ac312720ef122c69639d42b42f70f50e30\": rpc error: code = NotFound desc = could not find container \"f7e299e6246c0c0adf5bd2ee7b4d21ac312720ef122c69639d42b42f70f50e30\": container with ID starting with f7e299e6246c0c0adf5bd2ee7b4d21ac312720ef122c69639d42b42f70f50e30 not found: ID does not exist"
Sep 09 20:32:44.473812 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.473793    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"]
Sep 09 20:32:44.478455 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.478431    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-812760-openshift-default-5cd46b49c8-725xh"]
Sep 09 20:32:44.730183 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.730148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:44.789822 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.789788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:44.890225 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:44.890186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:45.122896 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.122817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:45.230301 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.230260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:45.314644 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.314613    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" path="/var/lib/kubelet/pods/69d67d76-d49b-451c-93ab-a9e53afda094/volumes"
Sep 09 20:32:45.332130 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.332095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:45.388716 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.388641    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"]
Sep 09 20:32:45.388987 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.388972    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e" containerName="payload-pre-processing"
Sep 09 20:32:45.389032 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.388990    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e" containerName="payload-pre-processing"
Sep 09 20:32:45.389032 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.389002    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" containerName="istio-proxy"
Sep 09 20:32:45.389032 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.389007    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" containerName="istio-proxy"
Sep 09 20:32:45.389032 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.389014    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac31df38-189a-45c1-8ada-fc80d8839f98" containerName="revoke-keys"
Sep 09 20:32:45.389032 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.389019    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac31df38-189a-45c1-8ada-fc80d8839f98" containerName="revoke-keys"
Sep 09 20:32:45.389225 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.389033    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" containerName="istio-proxy"
Sep 09 20:32:45.389225 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.389041    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" containerName="istio-proxy"
Sep 09 20:32:45.389225 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.389053    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7613f0db-2111-45b4-814f-71c4ffd40bcf" containerName="payload-processing"
Sep 09 20:32:45.389225 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.389059    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7613f0db-2111-45b4-814f-71c4ffd40bcf" containerName="payload-processing"
Sep 09 20:32:45.389225 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.389107    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac31df38-189a-45c1-8ada-fc80d8839f98" containerName="revoke-keys"
Sep 09 20:32:45.389225 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.389115    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4dfe886-e8cc-4d22-af9a-ff4dc75f6e4e" containerName="payload-pre-processing"
Sep 09 20:32:45.389225 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.389121    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" containerName="istio-proxy"
Sep 09 20:32:45.389225 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.389126    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d67d76-d49b-451c-93ab-a9e53afda094" containerName="istio-proxy"
Sep 09 20:32:45.389225 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.389132    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7613f0db-2111-45b4-814f-71c4ffd40bcf" containerName="payload-processing"
Sep 09 20:32:45.393562 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.393542    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"
Sep 09 20:32:45.396030 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.396004    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-08bc01-dockercfg-pdzsj\""
Sep 09 20:32:45.396164 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.396008    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-08bc01\""
Sep 09 20:32:45.401370 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.401341    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"]
Sep 09 20:32:45.415518 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.415477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:45.475674 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.475638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e8b805-ef19-4419-84ba-1aa247f38d2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w\" (UID: \"24e8b805-ef19-4419-84ba-1aa247f38d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"
Sep 09 20:32:45.475881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.475733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdfkz\" (UniqueName: \"kubernetes.io/projected/24e8b805-ef19-4419-84ba-1aa247f38d2f-kube-api-access-vdfkz\") pod \"payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w\" (UID: \"24e8b805-ef19-4419-84ba-1aa247f38d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"
Sep 09 20:32:45.508667 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.508632    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"]
Sep 09 20:32:45.511951 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.511934    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"
Sep 09 20:32:45.522651 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.522625    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"]
Sep 09 20:32:45.576321 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.576282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfdj7\" (UniqueName: \"kubernetes.io/projected/4a1a27b1-7285-4de8-956c-975e3b9260e2-kube-api-access-nfdj7\") pod \"payload-processing-e2e-explicit-08bc01-85748b97-zrx24\" (UID: \"4a1a27b1-7285-4de8-956c-975e3b9260e2\") " pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"
Sep 09 20:32:45.576321 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.576329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1a27b1-7285-4de8-956c-975e3b9260e2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-08bc01-85748b97-zrx24\" (UID: \"4a1a27b1-7285-4de8-956c-975e3b9260e2\") " pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"
Sep 09 20:32:45.576601 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.576408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdfkz\" (UniqueName: \"kubernetes.io/projected/24e8b805-ef19-4419-84ba-1aa247f38d2f-kube-api-access-vdfkz\") pod \"payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w\" (UID: \"24e8b805-ef19-4419-84ba-1aa247f38d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"
Sep 09 20:32:45.576601 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.576497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e8b805-ef19-4419-84ba-1aa247f38d2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w\" (UID: \"24e8b805-ef19-4419-84ba-1aa247f38d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"
Sep 09 20:32:45.577217 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.577197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e8b805-ef19-4419-84ba-1aa247f38d2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w\" (UID: \"24e8b805-ef19-4419-84ba-1aa247f38d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"
Sep 09 20:32:45.585160 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.585137    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdfkz\" (UniqueName: \"kubernetes.io/projected/24e8b805-ef19-4419-84ba-1aa247f38d2f-kube-api-access-vdfkz\") pod \"payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w\" (UID: \"24e8b805-ef19-4419-84ba-1aa247f38d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"
Sep 09 20:32:45.677497 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.677406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfdj7\" (UniqueName: \"kubernetes.io/projected/4a1a27b1-7285-4de8-956c-975e3b9260e2-kube-api-access-nfdj7\") pod \"payload-processing-e2e-explicit-08bc01-85748b97-zrx24\" (UID: \"4a1a27b1-7285-4de8-956c-975e3b9260e2\") " pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"
Sep 09 20:32:45.677497 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.677449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1a27b1-7285-4de8-956c-975e3b9260e2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-08bc01-85748b97-zrx24\" (UID: \"4a1a27b1-7285-4de8-956c-975e3b9260e2\") " pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"
Sep 09 20:32:45.678120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.678100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1a27b1-7285-4de8-956c-975e3b9260e2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-08bc01-85748b97-zrx24\" (UID: \"4a1a27b1-7285-4de8-956c-975e3b9260e2\") " pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"
Sep 09 20:32:45.685347 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.685327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfdj7\" (UniqueName: \"kubernetes.io/projected/4a1a27b1-7285-4de8-956c-975e3b9260e2-kube-api-access-nfdj7\") pod \"payload-processing-e2e-explicit-08bc01-85748b97-zrx24\" (UID: \"4a1a27b1-7285-4de8-956c-975e3b9260e2\") " pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"
Sep 09 20:32:45.706013 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.705989    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"
Sep 09 20:32:45.822304 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.822263    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"
Sep 09 20:32:45.859117 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.859084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:45.976439 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:45.976417    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"]
Sep 09 20:32:45.979134 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:32:45.979109    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e8b805_ef19_4419_84ba_1aa247f38d2f.slice/crio-a672b8b50c5b26a41d578b510c7869f4e49e05f9d41fbf0fab38b4b1a1d981f8 WatchSource:0}: Error finding container a672b8b50c5b26a41d578b510c7869f4e49e05f9d41fbf0fab38b4b1a1d981f8: Status 404 returned error can't find the container with id a672b8b50c5b26a41d578b510c7869f4e49e05f9d41fbf0fab38b4b1a1d981f8
Sep 09 20:32:46.095378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:46.095342    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"]
Sep 09 20:32:46.098542 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:32:46.098518    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1a27b1_7285_4de8_956c_975e3b9260e2.slice/crio-e49bc00558097ccd3a9b4ee4f5daf1d55b3b744a05d03fc432d6f376cb176670 WatchSource:0}: Error finding container e49bc00558097ccd3a9b4ee4f5daf1d55b3b744a05d03fc432d6f376cb176670: Status 404 returned error can't find the container with id e49bc00558097ccd3a9b4ee4f5daf1d55b3b744a05d03fc432d6f376cb176670
Sep 09 20:32:46.467649 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:46.467614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24" event={"ID":"4a1a27b1-7285-4de8-956c-975e3b9260e2","Type":"ContainerStarted","Data":"c28ae93e6ea3f32af3ec9da4e006406e0f972b6dc1e0865341259f79d23ec70e"}
Sep 09 20:32:46.468247 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:46.467674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24" event={"ID":"4a1a27b1-7285-4de8-956c-975e3b9260e2","Type":"ContainerStarted","Data":"e49bc00558097ccd3a9b4ee4f5daf1d55b3b744a05d03fc432d6f376cb176670"}
Sep 09 20:32:46.468247 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:46.467729    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"
Sep 09 20:32:46.469292 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:46.469265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w" event={"ID":"24e8b805-ef19-4419-84ba-1aa247f38d2f","Type":"ContainerStarted","Data":"658ca21ea2cae6450f474ef5edd8ad1ea6db3dc2a568170ee3ef96cbbc7857d1"}
Sep 09 20:32:46.469428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:46.469303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w" event={"ID":"24e8b805-ef19-4419-84ba-1aa247f38d2f","Type":"ContainerStarted","Data":"a672b8b50c5b26a41d578b510c7869f4e49e05f9d41fbf0fab38b4b1a1d981f8"}
Sep 09 20:32:46.469428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:46.469335    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"
Sep 09 20:32:46.502505 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:46.502455    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24" podStartSLOduration=1.502438181 podStartE2EDuration="1.502438181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:46.50167527 +0000 UTC m=+973.740777568" watchObservedRunningTime="2026-09-09 20:32:46.502438181 +0000 UTC m=+973.741540490"
Sep 09 20:32:57.476768 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:57.476678    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"
Sep 09 20:32:57.477141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:57.476758    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"
Sep 09 20:32:57.495537 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:57.495485    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w" podStartSLOduration=12.495470574 podStartE2EDuration="12.495470574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:46.530315564 +0000 UTC m=+973.769417865" watchObservedRunningTime="2026-09-09 20:32:57.495470574 +0000 UTC m=+984.734572868"
Sep 09 20:32:58.732092 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:58.732059    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"]
Sep 09 20:32:58.754910 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:58.754878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:58.901951 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:58.901918    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:32:59.248755 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:32:59.248710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:07.090997 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:07.090959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:07.774565 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:07.774524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:08.375649 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:08.375611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:08.476838 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:08.476801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:09.328356 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:09.328317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:11.873071 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:11.873027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:13.298263 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:13.298209    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:33:13.753025 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:13.752981    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk" podUID="290ca516-9aa2-48ed-924b-b812e2656b11" containerName="main" containerID="cri-o://e99566f481d844887a176b0380489d2075b6bdb8d3312617fbb8e23aaa46928e" gracePeriod=45
Sep 09 20:33:14.009932 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.009853    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:33:14.129513 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.129477    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-dshm\") pod \"290ca516-9aa2-48ed-924b-b812e2656b11\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") "
Sep 09 20:33:14.129749 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.129542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggh9r\" (UniqueName: \"kubernetes.io/projected/290ca516-9aa2-48ed-924b-b812e2656b11-kube-api-access-ggh9r\") pod \"290ca516-9aa2-48ed-924b-b812e2656b11\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") "
Sep 09 20:33:14.129749 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.129567    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-tmp-dir\") pod \"290ca516-9aa2-48ed-924b-b812e2656b11\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") "
Sep 09 20:33:14.129749 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.129642    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/290ca516-9aa2-48ed-924b-b812e2656b11-tls-certs\") pod \"290ca516-9aa2-48ed-924b-b812e2656b11\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") "
Sep 09 20:33:14.129749 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.129683    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-model-cache\") pod \"290ca516-9aa2-48ed-924b-b812e2656b11\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") "
Sep 09 20:33:14.129749 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.129704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-home\") pod \"290ca516-9aa2-48ed-924b-b812e2656b11\" (UID: \"290ca516-9aa2-48ed-924b-b812e2656b11\") "
Sep 09 20:33:14.130038 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.129898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "290ca516-9aa2-48ed-924b-b812e2656b11" (UID: "290ca516-9aa2-48ed-924b-b812e2656b11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:14.130038 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.129975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-home" (OuterVolumeSpecName: "home") pod "290ca516-9aa2-48ed-924b-b812e2656b11" (UID: "290ca516-9aa2-48ed-924b-b812e2656b11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:14.130038 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.130014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-model-cache" (OuterVolumeSpecName: "model-cache") pod "290ca516-9aa2-48ed-924b-b812e2656b11" (UID: "290ca516-9aa2-48ed-924b-b812e2656b11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:14.131866 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.131844    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290ca516-9aa2-48ed-924b-b812e2656b11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "290ca516-9aa2-48ed-924b-b812e2656b11" (UID: "290ca516-9aa2-48ed-924b-b812e2656b11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:33:14.131957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.131885    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-dshm" (OuterVolumeSpecName: "dshm") pod "290ca516-9aa2-48ed-924b-b812e2656b11" (UID: "290ca516-9aa2-48ed-924b-b812e2656b11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:14.132013 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.131964    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290ca516-9aa2-48ed-924b-b812e2656b11-kube-api-access-ggh9r" (OuterVolumeSpecName: "kube-api-access-ggh9r") pod "290ca516-9aa2-48ed-924b-b812e2656b11" (UID: "290ca516-9aa2-48ed-924b-b812e2656b11"). InnerVolumeSpecName "kube-api-access-ggh9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:33:14.231356 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.231316    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-home\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:33:14.231356 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.231349    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-dshm\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:33:14.231356 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.231358    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggh9r\" (UniqueName: \"kubernetes.io/projected/290ca516-9aa2-48ed-924b-b812e2656b11-kube-api-access-ggh9r\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:33:14.231356 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.231368    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-tmp-dir\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:33:14.231650 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.231377    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/290ca516-9aa2-48ed-924b-b812e2656b11-tls-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:33:14.231650 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.231385    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/290ca516-9aa2-48ed-924b-b812e2656b11-model-cache\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:33:14.608318 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.608283    2563 generic.go:356] "Generic (PLEG): container finished" podID="290ca516-9aa2-48ed-924b-b812e2656b11" containerID="e99566f481d844887a176b0380489d2075b6bdb8d3312617fbb8e23aaa46928e" exitCode=0
Sep 09 20:33:14.608766 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.608326    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"
Sep 09 20:33:14.608766 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.608341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk" event={"ID":"290ca516-9aa2-48ed-924b-b812e2656b11","Type":"ContainerDied","Data":"e99566f481d844887a176b0380489d2075b6bdb8d3312617fbb8e23aaa46928e"}
Sep 09 20:33:14.608766 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.608374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk" event={"ID":"290ca516-9aa2-48ed-924b-b812e2656b11","Type":"ContainerDied","Data":"5b288d2e9396f4526ebe088c5847abf18de92c1fbdac32348f02b9b591da34dd"}
Sep 09 20:33:14.608766 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.608391    2563 scope.go:117] "RemoveContainer" containerID="e99566f481d844887a176b0380489d2075b6bdb8d3312617fbb8e23aaa46928e"
Sep 09 20:33:14.617049 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.617028    2563 scope.go:117] "RemoveContainer" containerID="e99566f481d844887a176b0380489d2075b6bdb8d3312617fbb8e23aaa46928e"
Sep 09 20:33:14.617323 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:33:14.617306    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99566f481d844887a176b0380489d2075b6bdb8d3312617fbb8e23aaa46928e\": container with ID starting with e99566f481d844887a176b0380489d2075b6bdb8d3312617fbb8e23aaa46928e not found: ID does not exist" containerID="e99566f481d844887a176b0380489d2075b6bdb8d3312617fbb8e23aaa46928e"
Sep 09 20:33:14.617369 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.617334    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99566f481d844887a176b0380489d2075b6bdb8d3312617fbb8e23aaa46928e"} err="failed to get container status \"e99566f481d844887a176b0380489d2075b6bdb8d3312617fbb8e23aaa46928e\": rpc error: code = NotFound desc = could not find container \"e99566f481d844887a176b0380489d2075b6bdb8d3312617fbb8e23aaa46928e\": container with ID starting with e99566f481d844887a176b0380489d2075b6bdb8d3312617fbb8e23aaa46928e not found: ID does not exist"
Sep 09 20:33:14.629328 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.629302    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"]
Sep 09 20:33:14.633401 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:14.633378    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-836c5d71-kserve-558788f96d-m7dbk"]
Sep 09 20:33:15.313915 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:15.313883    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="290ca516-9aa2-48ed-924b-b812e2656b11" path="/var/lib/kubelet/pods/290ca516-9aa2-48ed-924b-b812e2656b11/volumes"
Sep 09 20:33:18.009519 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:18.009477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:18.125625 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:18.125561    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:18.242074 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:18.242041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:25.906070 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:25.906027    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:33:30.360875 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:30.360831    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:34.312099 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:34.312063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:40.207635 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:40.207594    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:33:40.713758 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:40.713709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:40.834262 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:40.834228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:40.921090 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:40.921045    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:43.977236 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:43.977199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:51.599477 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:51.599437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:52.500445 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:52.500408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:53.594720 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:53.594683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:33:59.350545 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:59.350505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:59.530484 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:59.530450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:59.633264 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:33:59.633165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:00.460397 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.460340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:00.560406 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.560370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:00.686458 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.686417    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:00.868508 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.868420    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"]
Sep 09 20:34:00.868773 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.868760    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="290ca516-9aa2-48ed-924b-b812e2656b11" containerName="main"
Sep 09 20:34:00.868823 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.868775    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="290ca516-9aa2-48ed-924b-b812e2656b11" containerName="main"
Sep 09 20:34:00.868858 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.868841    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="290ca516-9aa2-48ed-924b-b812e2656b11" containerName="main"
Sep 09 20:34:00.871699 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.871677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:00.880007 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.879981    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-08bc01\"/\"kube-root-ca.crt\""
Sep 09 20:34:00.880944 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.880918    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-08bc01\"/\"explicit-model-08bc01-kserve-self-signed-certs\""
Sep 09 20:34:00.881067 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.880950    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-08bc01\"/\"default-dockercfg-rhckc\""
Sep 09 20:34:00.881067 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.880968    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-08bc01\"/\"openshift-service-ca.crt\""
Sep 09 20:34:00.921030 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.920995    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"]
Sep 09 20:34:00.929900 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.929861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:00.951137 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.951096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-home\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:00.951137 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.951138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-dshm\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:00.951343 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.951164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-tmp-dir\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:00.951343 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.951234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-model-cache\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:00.951343 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.951274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-tls-certs\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:00.951343 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:00.951333    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqd4m\" (UniqueName: \"kubernetes.io/projected/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-kube-api-access-qqd4m\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.052274 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.052234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-model-cache\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.052462 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.052286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-tls-certs\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.052462 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.052317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqd4m\" (UniqueName: \"kubernetes.io/projected/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-kube-api-access-qqd4m\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.052462 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.052341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-home\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.052462 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.052361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-dshm\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.052462 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.052390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-tmp-dir\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.052856 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.052754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-model-cache\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.052856 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.052793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-home\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.052927 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.052867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-tmp-dir\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.054718 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.054693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-dshm\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.054866 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.054848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-tls-certs\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.076301 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.076271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqd4m\" (UniqueName: \"kubernetes.io/projected/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-kube-api-access-qqd4m\") pod \"explicit-model-08bc01-kserve-5d6cd465d9-67jxj\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") " pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.182053 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.181938    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:01.457109 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.457028    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"]
Sep 09 20:34:01.459856 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:34:01.459813    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eb9316b_4bd6_4a4b_a7c6_2f2cd29bb554.slice/crio-40a93dc7663fcfadd19df65980cafd1c02fb2d2f38ca4e9962f3cd452354466b WatchSource:0}: Error finding container 40a93dc7663fcfadd19df65980cafd1c02fb2d2f38ca4e9962f3cd452354466b: Status 404 returned error can't find the container with id 40a93dc7663fcfadd19df65980cafd1c02fb2d2f38ca4e9962f3cd452354466b
Sep 09 20:34:01.832338 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.832236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj" event={"ID":"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554","Type":"ContainerStarted","Data":"5365db96a73405e66e4cf76b61fe71cb12c79157d62a1ee9c6de9f1362e9ec26"}
Sep 09 20:34:01.832338 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.832282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj" event={"ID":"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554","Type":"ContainerStarted","Data":"40a93dc7663fcfadd19df65980cafd1c02fb2d2f38ca4e9962f3cd452354466b"}
Sep 09 20:34:01.851082 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:01.851023    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj" podStartSLOduration=1.851004493 podStartE2EDuration="1.851004493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:01.850706592 +0000 UTC m=+1049.089808894" watchObservedRunningTime="2026-09-09 20:34:01.851004493 +0000 UTC m=+1049.090106792"
Sep 09 20:34:02.410325 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:02.410287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:03.699538 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:03.699496    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:34:08.801205 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:08.801155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:08.914567 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:08.914524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:09.574883 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:09.574850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:11.182982 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:11.182938    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:11.183416 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:11.182996    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:11.195634 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:11.195601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:11.894464 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:11.894432    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:16.521910 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:16.521872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:16.621726 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:16.621689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:16.759183 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:16.759145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:17.185388 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.185351    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"]
Sep 09 20:34:17.188802 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.188783    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.191815 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.191786    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-36f068b3-openshift-default-dockercfg-z7jrz\""
Sep 09 20:34:17.193176 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.193143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-credential-socket\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.193309 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.193229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istiod-ca-cert\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.193309 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.193247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75hd\" (UniqueName: \"kubernetes.io/projected/5246aabf-1ade-4cb6-b509-88886a1a4ecd-kube-api-access-g75hd\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.193446 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.193346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-workload-certs\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.193446 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.193385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-envoy\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.193557 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.193453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-data\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.193557 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.193493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-workload-socket\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.193557 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.193517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-token\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.193724 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.193613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-podinfo\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.208403 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.208366    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"]
Sep 09 20:34:17.223374 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.223340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:17.294293 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.294256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-workload-certs\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.294293 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.294292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-envoy\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.294553 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.294313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-data\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.294553 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.294332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-workload-socket\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.294553 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.294351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-token\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.294553 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.294383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-podinfo\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.294553 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.294455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-credential-socket\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.294553 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.294510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istiod-ca-cert\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.294553 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.294533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75hd\" (UniqueName: \"kubernetes.io/projected/5246aabf-1ade-4cb6-b509-88886a1a4ecd-kube-api-access-g75hd\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.294913 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.294739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-workload-certs\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.294913 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.294784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-data\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.294913 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.294847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-workload-socket\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.295016 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.294998    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-credential-socket\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.295496 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.295472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istiod-ca-cert\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.296714 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.296691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-envoy\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.296985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.296966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-podinfo\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.305760 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.305733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75hd\" (UniqueName: \"kubernetes.io/projected/5246aabf-1ade-4cb6-b509-88886a1a4ecd-kube-api-access-g75hd\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.306320 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.306286    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-token\") pod \"e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") " pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.500190 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.500149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:17.800549 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.800374    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"]
Sep 09 20:34:17.803019 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:34:17.802987    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5246aabf_1ade_4cb6_b509_88886a1a4ecd.slice/crio-303cc4d671bd87eb97fef41474838cb6dbe70a2089183e988dc5ae7be6021cf0 WatchSource:0}: Error finding container 303cc4d671bd87eb97fef41474838cb6dbe70a2089183e988dc5ae7be6021cf0: Status 404 returned error can't find the container with id 303cc4d671bd87eb97fef41474838cb6dbe70a2089183e988dc5ae7be6021cf0
Sep 09 20:34:17.805256 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.805223    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:34:17.805339 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.805292    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:34:17.805339 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.805320    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:34:17.915236 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.915192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" event={"ID":"5246aabf-1ade-4cb6-b509-88886a1a4ecd","Type":"ContainerStarted","Data":"5349a80693b196631c859861b80287b52bc52a3613be3f91217b3dcb0249206f"}
Sep 09 20:34:17.915436 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.915311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" event={"ID":"5246aabf-1ade-4cb6-b509-88886a1a4ecd","Type":"ContainerStarted","Data":"303cc4d671bd87eb97fef41474838cb6dbe70a2089183e988dc5ae7be6021cf0"}
Sep 09 20:34:17.944696 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:17.944634    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" podStartSLOduration=0.944614973 podStartE2EDuration="944.614973ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:17.943219735 +0000 UTC m=+1065.182322062" watchObservedRunningTime="2026-09-09 20:34:17.944614973 +0000 UTC m=+1065.183717273"
Sep 09 20:34:18.303011 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:18.302969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:18.500646 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:18.500595    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:18.505834 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:18.505803    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:18.920637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:18.920603    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:18.921620 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:18.921601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:19.770863 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:19.770822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:20.477660 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:20.477618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:20.599846 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:20.599813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:20.762773 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:20.762667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:21.879130 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:21.879097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:22.793263 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:22.793224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:23.134919 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:23.134829    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:34:23.152737 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:23.152704    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"]
Sep 09 20:34:23.170785 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:23.170749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:23.841890 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:23.841849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:23.954689 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:23.954654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:24.042276 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.042240    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:24.529907 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.529873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:24.628786 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.628753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:24.642931 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.642894    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"]
Sep 09 20:34:24.646524 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.646501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"
Sep 09 20:34:24.649850 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.649830    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-36f068b3\""
Sep 09 20:34:24.657998 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.657973    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"]
Sep 09 20:34:24.661875 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.661849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0154e5b-130e-47a2-8545-b4a252955a21-plugins-config-volume\") pod \"payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg\" (UID: \"e0154e5b-130e-47a2-8545-b4a252955a21\") " pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"
Sep 09 20:34:24.661992 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.661917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pb4r\" (UniqueName: \"kubernetes.io/projected/e0154e5b-130e-47a2-8545-b4a252955a21-kube-api-access-9pb4r\") pod \"payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg\" (UID: \"e0154e5b-130e-47a2-8545-b4a252955a21\") " pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"
Sep 09 20:34:24.762490 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.762450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pb4r\" (UniqueName: \"kubernetes.io/projected/e0154e5b-130e-47a2-8545-b4a252955a21-kube-api-access-9pb4r\") pod \"payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg\" (UID: \"e0154e5b-130e-47a2-8545-b4a252955a21\") " pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"
Sep 09 20:34:24.762711 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.762594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0154e5b-130e-47a2-8545-b4a252955a21-plugins-config-volume\") pod \"payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg\" (UID: \"e0154e5b-130e-47a2-8545-b4a252955a21\") " pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"
Sep 09 20:34:24.763528 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.763495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0154e5b-130e-47a2-8545-b4a252955a21-plugins-config-volume\") pod \"payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg\" (UID: \"e0154e5b-130e-47a2-8545-b4a252955a21\") " pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"
Sep 09 20:34:24.773960 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.773927    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pb4r\" (UniqueName: \"kubernetes.io/projected/e0154e5b-130e-47a2-8545-b4a252955a21-kube-api-access-9pb4r\") pod \"payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg\" (UID: \"e0154e5b-130e-47a2-8545-b4a252955a21\") " pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"
Sep 09 20:34:24.958250 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:24.958216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"
Sep 09 20:34:25.014616 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:25.014567    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:25.234793 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:25.234767    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"]
Sep 09 20:34:25.237122 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:34:25.237095    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0154e5b_130e_47a2_8545_b4a252955a21.slice/crio-98ad41075210ad292d8265204984b94875b096176dcaa561b3b7c456fd53dc86 WatchSource:0}: Error finding container 98ad41075210ad292d8265204984b94875b096176dcaa561b3b7c456fd53dc86: Status 404 returned error can't find the container with id 98ad41075210ad292d8265204984b94875b096176dcaa561b3b7c456fd53dc86
Sep 09 20:34:25.960446 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:25.960408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg" event={"ID":"e0154e5b-130e-47a2-8545-b4a252955a21","Type":"ContainerStarted","Data":"89c051e42d13b467c996d16b8d7aa4dc5e545561f2b5f7c183bc159a3713ca38"}
Sep 09 20:34:25.960870 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:25.960471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg" event={"ID":"e0154e5b-130e-47a2-8545-b4a252955a21","Type":"ContainerStarted","Data":"98ad41075210ad292d8265204984b94875b096176dcaa561b3b7c456fd53dc86"}
Sep 09 20:34:25.960870 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:25.960522    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"
Sep 09 20:34:25.997276 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:25.997216    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg" podStartSLOduration=1.997201234 podStartE2EDuration="1.997201234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:25.994971694 +0000 UTC m=+1073.234073996" watchObservedRunningTime="2026-09-09 20:34:25.997201234 +0000 UTC m=+1073.236303579"
Sep 09 20:34:26.024035 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:26.023996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:26.853015 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:26.852967    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:27.790796 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:27.790760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:28.550734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:28.550690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:33.559755 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:33.559719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:34.299592 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.299538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:34.323746 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.323710    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"]
Sep 09 20:34:34.323983 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.323956    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24" podUID="4a1a27b1-7285-4de8-956c-975e3b9260e2" containerName="payload-processing" containerID="cri-o://c28ae93e6ea3f32af3ec9da4e006406e0f972b6dc1e0865341259f79d23ec70e" gracePeriod=30
Sep 09 20:34:34.333790 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.333759    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"]
Sep 09 20:34:34.333996 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.333964    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w" podUID="24e8b805-ef19-4419-84ba-1aa247f38d2f" containerName="payload-pre-processing" containerID="cri-o://658ca21ea2cae6450f474ef5edd8ad1ea6db3dc2a568170ee3ef96cbbc7857d1" gracePeriod=30
Sep 09 20:34:34.388028 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.387990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:34.452772 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.452504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:34.554296 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.554158    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:34.681886 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.681808    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"
Sep 09 20:34:34.696478 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.696432    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"
Sep 09 20:34:34.774074 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.774041    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e8b805-ef19-4419-84ba-1aa247f38d2f-plugins-config-volume\") pod \"24e8b805-ef19-4419-84ba-1aa247f38d2f\" (UID: \"24e8b805-ef19-4419-84ba-1aa247f38d2f\") "
Sep 09 20:34:34.774247 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.774104    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfdj7\" (UniqueName: \"kubernetes.io/projected/4a1a27b1-7285-4de8-956c-975e3b9260e2-kube-api-access-nfdj7\") pod \"4a1a27b1-7285-4de8-956c-975e3b9260e2\" (UID: \"4a1a27b1-7285-4de8-956c-975e3b9260e2\") "
Sep 09 20:34:34.774247 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.774153    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1a27b1-7285-4de8-956c-975e3b9260e2-plugins-config-volume\") pod \"4a1a27b1-7285-4de8-956c-975e3b9260e2\" (UID: \"4a1a27b1-7285-4de8-956c-975e3b9260e2\") "
Sep 09 20:34:34.774247 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.774199    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdfkz\" (UniqueName: \"kubernetes.io/projected/24e8b805-ef19-4419-84ba-1aa247f38d2f-kube-api-access-vdfkz\") pod \"24e8b805-ef19-4419-84ba-1aa247f38d2f\" (UID: \"24e8b805-ef19-4419-84ba-1aa247f38d2f\") "
Sep 09 20:34:34.774940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.774910    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1a27b1-7285-4de8-956c-975e3b9260e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4a1a27b1-7285-4de8-956c-975e3b9260e2" (UID: "4a1a27b1-7285-4de8-956c-975e3b9260e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:34.775015 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.774972    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e8b805-ef19-4419-84ba-1aa247f38d2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24e8b805-ef19-4419-84ba-1aa247f38d2f" (UID: "24e8b805-ef19-4419-84ba-1aa247f38d2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:34.776470 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.776443    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e8b805-ef19-4419-84ba-1aa247f38d2f-kube-api-access-vdfkz" (OuterVolumeSpecName: "kube-api-access-vdfkz") pod "24e8b805-ef19-4419-84ba-1aa247f38d2f" (UID: "24e8b805-ef19-4419-84ba-1aa247f38d2f"). InnerVolumeSpecName "kube-api-access-vdfkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:34.776566 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.776470    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1a27b1-7285-4de8-956c-975e3b9260e2-kube-api-access-nfdj7" (OuterVolumeSpecName: "kube-api-access-nfdj7") pod "4a1a27b1-7285-4de8-956c-975e3b9260e2" (UID: "4a1a27b1-7285-4de8-956c-975e3b9260e2"). InnerVolumeSpecName "kube-api-access-nfdj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:34.875084 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.874995    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfdj7\" (UniqueName: \"kubernetes.io/projected/4a1a27b1-7285-4de8-956c-975e3b9260e2-kube-api-access-nfdj7\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:34:34.875084 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.875024    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1a27b1-7285-4de8-956c-975e3b9260e2-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:34:34.875084 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.875033    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdfkz\" (UniqueName: \"kubernetes.io/projected/24e8b805-ef19-4419-84ba-1aa247f38d2f-kube-api-access-vdfkz\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:34:34.875084 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.875042    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e8b805-ef19-4419-84ba-1aa247f38d2f-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:34:34.933183 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:34.933142    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:35.017822 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.017777    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a1a27b1-7285-4de8-956c-975e3b9260e2" containerID="c28ae93e6ea3f32af3ec9da4e006406e0f972b6dc1e0865341259f79d23ec70e" exitCode=0
Sep 09 20:34:35.017822 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.017816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24" event={"ID":"4a1a27b1-7285-4de8-956c-975e3b9260e2","Type":"ContainerDied","Data":"c28ae93e6ea3f32af3ec9da4e006406e0f972b6dc1e0865341259f79d23ec70e"}
Sep 09 20:34:35.018109 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.017905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24" event={"ID":"4a1a27b1-7285-4de8-956c-975e3b9260e2","Type":"ContainerDied","Data":"e49bc00558097ccd3a9b4ee4f5daf1d55b3b744a05d03fc432d6f376cb176670"}
Sep 09 20:34:35.018109 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.017941    2563 scope.go:117] "RemoveContainer" containerID="c28ae93e6ea3f32af3ec9da4e006406e0f972b6dc1e0865341259f79d23ec70e"
Sep 09 20:34:35.018109 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.017978    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"
Sep 09 20:34:35.020697 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.020671    2563 generic.go:356] "Generic (PLEG): container finished" podID="24e8b805-ef19-4419-84ba-1aa247f38d2f" containerID="658ca21ea2cae6450f474ef5edd8ad1ea6db3dc2a568170ee3ef96cbbc7857d1" exitCode=0
Sep 09 20:34:35.020804 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.020735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w" event={"ID":"24e8b805-ef19-4419-84ba-1aa247f38d2f","Type":"ContainerDied","Data":"658ca21ea2cae6450f474ef5edd8ad1ea6db3dc2a568170ee3ef96cbbc7857d1"}
Sep 09 20:34:35.020804 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.020769    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w" event={"ID":"24e8b805-ef19-4419-84ba-1aa247f38d2f","Type":"ContainerDied","Data":"a672b8b50c5b26a41d578b510c7869f4e49e05f9d41fbf0fab38b4b1a1d981f8"}
Sep 09 20:34:35.020874 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.020819    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"
Sep 09 20:34:35.028880 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.028848    2563 scope.go:117] "RemoveContainer" containerID="c28ae93e6ea3f32af3ec9da4e006406e0f972b6dc1e0865341259f79d23ec70e"
Sep 09 20:34:35.029237 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:34:35.029208    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28ae93e6ea3f32af3ec9da4e006406e0f972b6dc1e0865341259f79d23ec70e\": container with ID starting with c28ae93e6ea3f32af3ec9da4e006406e0f972b6dc1e0865341259f79d23ec70e not found: ID does not exist" containerID="c28ae93e6ea3f32af3ec9da4e006406e0f972b6dc1e0865341259f79d23ec70e"
Sep 09 20:34:35.029347 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.029248    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28ae93e6ea3f32af3ec9da4e006406e0f972b6dc1e0865341259f79d23ec70e"} err="failed to get container status \"c28ae93e6ea3f32af3ec9da4e006406e0f972b6dc1e0865341259f79d23ec70e\": rpc error: code = NotFound desc = could not find container \"c28ae93e6ea3f32af3ec9da4e006406e0f972b6dc1e0865341259f79d23ec70e\": container with ID starting with c28ae93e6ea3f32af3ec9da4e006406e0f972b6dc1e0865341259f79d23ec70e not found: ID does not exist"
Sep 09 20:34:35.029347 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.029272    2563 scope.go:117] "RemoveContainer" containerID="658ca21ea2cae6450f474ef5edd8ad1ea6db3dc2a568170ee3ef96cbbc7857d1"
Sep 09 20:34:35.033554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.033480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:35.037975 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.037955    2563 scope.go:117] "RemoveContainer" containerID="658ca21ea2cae6450f474ef5edd8ad1ea6db3dc2a568170ee3ef96cbbc7857d1"
Sep 09 20:34:35.038278 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:34:35.038259    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658ca21ea2cae6450f474ef5edd8ad1ea6db3dc2a568170ee3ef96cbbc7857d1\": container with ID starting with 658ca21ea2cae6450f474ef5edd8ad1ea6db3dc2a568170ee3ef96cbbc7857d1 not found: ID does not exist" containerID="658ca21ea2cae6450f474ef5edd8ad1ea6db3dc2a568170ee3ef96cbbc7857d1"
Sep 09 20:34:35.038321 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.038288    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658ca21ea2cae6450f474ef5edd8ad1ea6db3dc2a568170ee3ef96cbbc7857d1"} err="failed to get container status \"658ca21ea2cae6450f474ef5edd8ad1ea6db3dc2a568170ee3ef96cbbc7857d1\": rpc error: code = NotFound desc = could not find container \"658ca21ea2cae6450f474ef5edd8ad1ea6db3dc2a568170ee3ef96cbbc7857d1\": container with ID starting with 658ca21ea2cae6450f474ef5edd8ad1ea6db3dc2a568170ee3ef96cbbc7857d1 not found: ID does not exist"
Sep 09 20:34:35.041716 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.041691    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"]
Sep 09 20:34:35.045482 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.045444    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-08bc01-85748b97-zrx24"]
Sep 09 20:34:35.056487 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.056442    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"]
Sep 09 20:34:35.061512 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.061469    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-08bc01-9dd57f8-6bl6w"]
Sep 09 20:34:35.313078 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.313042    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24e8b805-ef19-4419-84ba-1aa247f38d2f" path="/var/lib/kubelet/pods/24e8b805-ef19-4419-84ba-1aa247f38d2f/volumes"
Sep 09 20:34:35.313416 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.313403    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a1a27b1-7285-4de8-956c-975e3b9260e2" path="/var/lib/kubelet/pods/4a1a27b1-7285-4de8-956c-975e3b9260e2/volumes"
Sep 09 20:34:35.688764 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:35.688674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:36.680748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:36.680705    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:36.968225 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:36.968185    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"
Sep 09 20:34:37.480240 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:37.480201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:37.785516 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:37.785422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:38.152515 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.152414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:38.172914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.172869    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj" podUID="3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554" containerName="main" containerID="cri-o://5365db96a73405e66e4cf76b61fe71cb12c79157d62a1ee9c6de9f1362e9ec26" gracePeriod=45
Sep 09 20:34:38.454597 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.453752    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:38.613194 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.613157    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-dshm\") pod \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") "
Sep 09 20:34:38.613194 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.613203    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-home\") pod \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") "
Sep 09 20:34:38.613443 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.613241    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-tls-certs\") pod \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") "
Sep 09 20:34:38.613443 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.613281    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-tmp-dir\") pod \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") "
Sep 09 20:34:38.613443 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.613295    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-model-cache\") pod \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") "
Sep 09 20:34:38.613443 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.613322    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqd4m\" (UniqueName: \"kubernetes.io/projected/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-kube-api-access-qqd4m\") pod \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\" (UID: \"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554\") "
Sep 09 20:34:38.613701 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.613598    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-home" (OuterVolumeSpecName: "home") pod "3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554" (UID: "3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:38.613701 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.613610    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554" (UID: "3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:38.613701 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.613626    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-model-cache" (OuterVolumeSpecName: "model-cache") pod "3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554" (UID: "3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:38.615566 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.615528    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554" (UID: "3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:38.615709 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.615603    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-dshm" (OuterVolumeSpecName: "dshm") pod "3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554" (UID: "3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:38.615709 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.615646    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-kube-api-access-qqd4m" (OuterVolumeSpecName: "kube-api-access-qqd4m") pod "3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554" (UID: "3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554"). InnerVolumeSpecName "kube-api-access-qqd4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:38.714363 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.714333    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-home\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:34:38.714363 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.714359    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-tls-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:34:38.714554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.714371    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-tmp-dir\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:34:38.714554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.714380    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-model-cache\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:34:38.714554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.714389    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqd4m\" (UniqueName: \"kubernetes.io/projected/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-kube-api-access-qqd4m\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:34:38.714554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.714398    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554-dshm\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:34:38.815265 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:38.815227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:39.045131 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:39.045036    2563 generic.go:356] "Generic (PLEG): container finished" podID="3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554" containerID="5365db96a73405e66e4cf76b61fe71cb12c79157d62a1ee9c6de9f1362e9ec26" exitCode=0
Sep 09 20:34:39.045131 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:39.045078    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"
Sep 09 20:34:39.045360 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:39.045134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj" event={"ID":"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554","Type":"ContainerDied","Data":"5365db96a73405e66e4cf76b61fe71cb12c79157d62a1ee9c6de9f1362e9ec26"}
Sep 09 20:34:39.045360 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:39.045186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj" event={"ID":"3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554","Type":"ContainerDied","Data":"40a93dc7663fcfadd19df65980cafd1c02fb2d2f38ca4e9962f3cd452354466b"}
Sep 09 20:34:39.045360 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:39.045210    2563 scope.go:117] "RemoveContainer" containerID="5365db96a73405e66e4cf76b61fe71cb12c79157d62a1ee9c6de9f1362e9ec26"
Sep 09 20:34:39.054244 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:39.054223    2563 scope.go:117] "RemoveContainer" containerID="5365db96a73405e66e4cf76b61fe71cb12c79157d62a1ee9c6de9f1362e9ec26"
Sep 09 20:34:39.054634 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:34:39.054611    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5365db96a73405e66e4cf76b61fe71cb12c79157d62a1ee9c6de9f1362e9ec26\": container with ID starting with 5365db96a73405e66e4cf76b61fe71cb12c79157d62a1ee9c6de9f1362e9ec26 not found: ID does not exist" containerID="5365db96a73405e66e4cf76b61fe71cb12c79157d62a1ee9c6de9f1362e9ec26"
Sep 09 20:34:39.054734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:39.054646    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5365db96a73405e66e4cf76b61fe71cb12c79157d62a1ee9c6de9f1362e9ec26"} err="failed to get container status \"5365db96a73405e66e4cf76b61fe71cb12c79157d62a1ee9c6de9f1362e9ec26\": rpc error: code = NotFound desc = could not find container \"5365db96a73405e66e4cf76b61fe71cb12c79157d62a1ee9c6de9f1362e9ec26\": container with ID starting with 5365db96a73405e66e4cf76b61fe71cb12c79157d62a1ee9c6de9f1362e9ec26 not found: ID does not exist"
Sep 09 20:34:39.066037 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:39.066008    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"]
Sep 09 20:34:39.069295 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:39.069269    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-08bc01/explicit-model-08bc01-kserve-5d6cd465d9-67jxj"]
Sep 09 20:34:39.321091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:39.321011    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554" path="/var/lib/kubelet/pods/3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554/volumes"
Sep 09 20:34:39.535354 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:39.535302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:40.413755 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:40.413711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:41.127158 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:41.127113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:41.231705 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:41.231644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:41.652242 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:41.652203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:41.764538 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:41.764504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:41.861262 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:41.861228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:42.065944 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.065910    2563 generic.go:356] "Generic (PLEG): container finished" podID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerID="5349a80693b196631c859861b80287b52bc52a3613be3f91217b3dcb0249206f" exitCode=0
Sep 09 20:34:42.066123 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.065964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" event={"ID":"5246aabf-1ade-4cb6-b509-88886a1a4ecd","Type":"ContainerDied","Data":"5349a80693b196631c859861b80287b52bc52a3613be3f91217b3dcb0249206f"}
Sep 09 20:34:42.066399 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.066378    2563 scope.go:117] "RemoveContainer" containerID="5349a80693b196631c859861b80287b52bc52a3613be3f91217b3dcb0249206f"
Sep 09 20:34:42.067858 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.067797    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:34:42.067986 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.067897    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:34:42.067986 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.067946    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:34:42.088617 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.088372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:42.569282 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.569246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:42.886656 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.886556    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"]
Sep 09 20:34:42.887175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.886924    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e8b805-ef19-4419-84ba-1aa247f38d2f" containerName="payload-pre-processing"
Sep 09 20:34:42.887175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.886937    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e8b805-ef19-4419-84ba-1aa247f38d2f" containerName="payload-pre-processing"
Sep 09 20:34:42.887175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.886956    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1a27b1-7285-4de8-956c-975e3b9260e2" containerName="payload-processing"
Sep 09 20:34:42.887175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.886964    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1a27b1-7285-4de8-956c-975e3b9260e2" containerName="payload-processing"
Sep 09 20:34:42.887175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.886982    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554" containerName="main"
Sep 09 20:34:42.887175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.886991    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554" containerName="main"
Sep 09 20:34:42.887175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.887067    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb9316b-4bd6-4a4b-a7c6-2f2cd29bb554" containerName="main"
Sep 09 20:34:42.887175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.887082    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1a27b1-7285-4de8-956c-975e3b9260e2" containerName="payload-processing"
Sep 09 20:34:42.887175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.887093    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="24e8b805-ef19-4419-84ba-1aa247f38d2f" containerName="payload-pre-processing"
Sep 09 20:34:42.889661 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.889602    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:42.894226 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.894205    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-82f543-openshift-default-dockercfg-rslp6\""
Sep 09 20:34:42.903418 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.903387    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"]
Sep 09 20:34:42.918106 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.918075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:42.955166 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.955116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-podinfo\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:42.955345 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.955172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-workload-certs\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:42.955345 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.955254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-workload-socket\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:42.955438 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.955348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-envoy\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:42.955476 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.955454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istiod-ca-cert\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:42.955517 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.955488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-data\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:42.955552 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.955513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-token\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:42.955624 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.955601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-credential-socket\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:42.955678 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:42.955642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kbkp\" (UniqueName: \"kubernetes.io/projected/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-kube-api-access-5kbkp\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.056614 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.056547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istiod-ca-cert\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.056823 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.056626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-data\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.056823 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.056657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-token\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.056823 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.056690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-credential-socket\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.056823 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.056719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kbkp\" (UniqueName: \"kubernetes.io/projected/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-kube-api-access-5kbkp\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.056823 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.056769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-podinfo\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.056823 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.056798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-workload-certs\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.057152 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.056847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-workload-socket\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.057152 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.056894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-envoy\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.057774 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.057741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-credential-socket\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.057774 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.057751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-workload-certs\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.057988 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.057963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-data\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.058322 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.058297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-workload-socket\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.058633 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.058609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istiod-ca-cert\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.063628 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.061850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-envoy\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.069889 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.069037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-podinfo\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.071277 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.070371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-token\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.071277 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.071204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kbkp\" (UniqueName: \"kubernetes.io/projected/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-kube-api-access-5kbkp\") pod \"e2e-noten-82f543-openshift-default-746d47c544-g7vn6\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") " pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.076663 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.076626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" event={"ID":"5246aabf-1ade-4cb6-b509-88886a1a4ecd","Type":"ContainerStarted","Data":"d321909f1d05c52db1cfbdb1223238e04e3fefd767ada8caf9bd68a2bb6af580"}
Sep 09 20:34:43.096935 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.096896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:43.202012 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.201914    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:43.500024 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.500000    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"]
Sep 09 20:34:43.500958 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.500933    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:43.501163 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.501141    2563 patch_prober.go:28] interesting pod/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 09 20:34:43.501261 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.501194    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 09 20:34:43.501749 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:34:43.501725    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ce2c22_df27_47db_a0ca_0e22ae1ca5dd.slice/crio-aa4c3ea2d7f1785c65e7181bd803b51843a40df52393a573903d869d5ca0c7b7 WatchSource:0}: Error finding container aa4c3ea2d7f1785c65e7181bd803b51843a40df52393a573903d869d5ca0c7b7: Status 404 returned error can't find the container with id aa4c3ea2d7f1785c65e7181bd803b51843a40df52393a573903d869d5ca0c7b7
Sep 09 20:34:43.503999 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.503972    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:34:43.504072 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.504034    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:34:43.504072 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:43.504058    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:34:44.089353 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:44.089314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6" event={"ID":"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd","Type":"ContainerStarted","Data":"a62849749085bdcab01244f9f922fa713b2b4954bfb15d569fa7114e8f3ce1b8"}
Sep 09 20:34:44.089846 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:44.089392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6" event={"ID":"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd","Type":"ContainerStarted","Data":"aa4c3ea2d7f1785c65e7181bd803b51843a40df52393a573903d869d5ca0c7b7"}
Sep 09 20:34:44.110353 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:44.110300    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6" podStartSLOduration=2.110284777 podStartE2EDuration="2.110284777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:44.107834787 +0000 UTC m=+1091.346937097" watchObservedRunningTime="2026-09-09 20:34:44.110284777 +0000 UTC m=+1091.349387134"
Sep 09 20:34:44.202156 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:44.202115    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:44.203919 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:44.203887    2563 patch_prober.go:28] interesting pod/e2e-noten-82f543-openshift-default-746d47c544-g7vn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 09 20:34:44.204052 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:44.203962    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6" podUID="29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 09 20:34:44.414296 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:44.414210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:44.501417 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:44.501379    2563 patch_prober.go:28] interesting pod/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 09 20:34:44.501619 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:44.501446    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 09 20:34:44.506959 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:44.506931    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:34:44.510355 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:44.510326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:44.675381 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:44.675276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:45.130975 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:45.130922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:45.207359 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:45.207323    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:45.500660 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:45.500623    2563 patch_prober.go:28] interesting pod/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 09 20:34:45.500834 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:45.500680    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 09 20:34:46.099946 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:46.099910    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:46.101001 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:46.100978    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:34:46.500866 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:46.500825    2563 patch_prober.go:28] interesting pod/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 09 20:34:46.501259 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:46.500901    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 09 20:34:47.500950 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:47.500914    2563 patch_prober.go:28] interesting pod/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 09 20:34:47.501334 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:47.501009    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 09 20:34:48.501358 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:48.501320    2563 patch_prober.go:28] interesting pod/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 09 20:34:48.501779 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:48.501382    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 09 20:34:48.843173 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:48.843081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:48.921231 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:48.921191    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:48.957141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:48.957108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:49.505751 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:49.505716    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:49.506895 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:49.506873    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:34:50.381094 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:50.381055    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:51.433063 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:51.433012    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:51.550052 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:51.550014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:52.560083 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:52.560028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:53.583677 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:34:53.583638    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:02.264957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:02.264919    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:02.293469 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:02.293429    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"]
Sep 09 20:35:02.293992 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:02.293948    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg" podUID="e0154e5b-130e-47a2-8545-b4a252955a21" containerName="payload-processing" containerID="cri-o://89c051e42d13b467c996d16b8d7aa4dc5e545561f2b5f7c183bc159a3713ca38" gracePeriod=30
Sep 09 20:35:02.360588 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:02.360534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:02.460646 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:02.460606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:02.546397 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:02.546370    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"
Sep 09 20:35:02.565968 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:02.565927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:02.650425 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:02.650387    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0154e5b-130e-47a2-8545-b4a252955a21-plugins-config-volume\") pod \"e0154e5b-130e-47a2-8545-b4a252955a21\" (UID: \"e0154e5b-130e-47a2-8545-b4a252955a21\") "
Sep 09 20:35:02.650652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:02.650450    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pb4r\" (UniqueName: \"kubernetes.io/projected/e0154e5b-130e-47a2-8545-b4a252955a21-kube-api-access-9pb4r\") pod \"e0154e5b-130e-47a2-8545-b4a252955a21\" (UID: \"e0154e5b-130e-47a2-8545-b4a252955a21\") "
Sep 09 20:35:02.650940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:02.650833    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0154e5b-130e-47a2-8545-b4a252955a21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0154e5b-130e-47a2-8545-b4a252955a21" (UID: "e0154e5b-130e-47a2-8545-b4a252955a21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:02.653995 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:02.653892    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0154e5b-130e-47a2-8545-b4a252955a21-kube-api-access-9pb4r" (OuterVolumeSpecName: "kube-api-access-9pb4r") pod "e0154e5b-130e-47a2-8545-b4a252955a21" (UID: "e0154e5b-130e-47a2-8545-b4a252955a21"). InnerVolumeSpecName "kube-api-access-9pb4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:02.752044 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:02.752001    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0154e5b-130e-47a2-8545-b4a252955a21-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:02.752044 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:02.752040    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pb4r\" (UniqueName: \"kubernetes.io/projected/e0154e5b-130e-47a2-8545-b4a252955a21-kube-api-access-9pb4r\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.008128 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:03.008088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:03.103822 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:03.103788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:03.187555 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:03.187520    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0154e5b-130e-47a2-8545-b4a252955a21" containerID="89c051e42d13b467c996d16b8d7aa4dc5e545561f2b5f7c183bc159a3713ca38" exitCode=0
Sep 09 20:35:03.187764 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:03.187563    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"
Sep 09 20:35:03.187764 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:03.187607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg" event={"ID":"e0154e5b-130e-47a2-8545-b4a252955a21","Type":"ContainerDied","Data":"89c051e42d13b467c996d16b8d7aa4dc5e545561f2b5f7c183bc159a3713ca38"}
Sep 09 20:35:03.187764 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:03.187669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg" event={"ID":"e0154e5b-130e-47a2-8545-b4a252955a21","Type":"ContainerDied","Data":"98ad41075210ad292d8265204984b94875b096176dcaa561b3b7c456fd53dc86"}
Sep 09 20:35:03.187764 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:03.187699    2563 scope.go:117] "RemoveContainer" containerID="89c051e42d13b467c996d16b8d7aa4dc5e545561f2b5f7c183bc159a3713ca38"
Sep 09 20:35:03.197104 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:03.197082    2563 scope.go:117] "RemoveContainer" containerID="89c051e42d13b467c996d16b8d7aa4dc5e545561f2b5f7c183bc159a3713ca38"
Sep 09 20:35:03.197401 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:35:03.197383    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c051e42d13b467c996d16b8d7aa4dc5e545561f2b5f7c183bc159a3713ca38\": container with ID starting with 89c051e42d13b467c996d16b8d7aa4dc5e545561f2b5f7c183bc159a3713ca38 not found: ID does not exist" containerID="89c051e42d13b467c996d16b8d7aa4dc5e545561f2b5f7c183bc159a3713ca38"
Sep 09 20:35:03.197447 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:03.197413    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c051e42d13b467c996d16b8d7aa4dc5e545561f2b5f7c183bc159a3713ca38"} err="failed to get container status \"89c051e42d13b467c996d16b8d7aa4dc5e545561f2b5f7c183bc159a3713ca38\": rpc error: code = NotFound desc = could not find container \"89c051e42d13b467c996d16b8d7aa4dc5e545561f2b5f7c183bc159a3713ca38\": container with ID starting with 89c051e42d13b467c996d16b8d7aa4dc5e545561f2b5f7c183bc159a3713ca38 not found: ID does not exist"
Sep 09 20:35:03.209796 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:03.209758    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"]
Sep 09 20:35:03.213605 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:03.213565    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-36f068b3-7db479b649-zp4dg"]
Sep 09 20:35:03.313004 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:03.312922    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0154e5b-130e-47a2-8545-b4a252955a21" path="/var/lib/kubelet/pods/e0154e5b-130e-47a2-8545-b4a252955a21/volumes"
Sep 09 20:35:03.927960 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:03.927916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:04.958491 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:04.958449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:06.476988 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:06.476954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:06.592783 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:06.592749    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"]
Sep 09 20:35:06.592978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:06.592955    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6" podUID="29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" containerName="istio-proxy" containerID="cri-o://a62849749085bdcab01244f9f922fa713b2b4954bfb15d569fa7114e8f3ce1b8" gracePeriod=30
Sep 09 20:35:06.620637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:06.620601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:06.882211 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:06.882125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:07.000270 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:07.000236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:07.215196 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:07.215163    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:35:08.838152 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:08.838118    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:08.933125 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:08.933091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:09.134069 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:09.133991    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"]
Sep 09 20:35:09.134271 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:09.134243    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerName="istio-proxy" containerID="cri-o://d321909f1d05c52db1cfbdb1223238e04e3fefd767ada8caf9bd68a2bb6af580" gracePeriod=30
Sep 09 20:35:09.170183 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:09.170145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:09.594394 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:09.594356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:09.699233 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:09.699197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:09.840991 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:09.840947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:11.843198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.843171    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:35:11.934700 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.934607    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-credential-socket\") pod \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") "
Sep 09 20:35:11.934700 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.934647    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-workload-socket\") pod \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") "
Sep 09 20:35:11.934700 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.934692    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-data\") pod \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") "
Sep 09 20:35:11.934932 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.934730    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istiod-ca-cert\") pod \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") "
Sep 09 20:35:11.934932 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.934753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-token\") pod \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") "
Sep 09 20:35:11.934932 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.934809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-envoy\") pod \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") "
Sep 09 20:35:11.934932 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.934887    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-podinfo\") pod \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") "
Sep 09 20:35:11.934932 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.934910    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-workload-certs\") pod \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") "
Sep 09 20:35:11.935194 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.934934    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kbkp\" (UniqueName: \"kubernetes.io/projected/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-kube-api-access-5kbkp\") pod \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\" (UID: \"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd\") "
Sep 09 20:35:11.935194 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.934938    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" (UID: "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:11.935194 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.935047    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" (UID: "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:11.935194 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.935063    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" (UID: "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:11.935194 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.935178    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" (UID: "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:11.935520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.935205    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" (UID: "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:11.935520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.935366    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:11.935520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.935387    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:11.935520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.935402    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:11.935520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.935415    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:11.935520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.935427    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:11.937086 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.937046    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-kube-api-access-5kbkp" (OuterVolumeSpecName: "kube-api-access-5kbkp") pod "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" (UID: "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd"). InnerVolumeSpecName "kube-api-access-5kbkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:11.937221 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.937139    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" (UID: "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:11.937221 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.937166    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" (UID: "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:11.937221 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:11.937206    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" (UID: "29ce2c22-df27-47db-a0ca-0e22ae1ca5dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:35:12.036477 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.036438    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:12.036477 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.036471    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:12.036715 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.036487    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kbkp\" (UniqueName: \"kubernetes.io/projected/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-kube-api-access-5kbkp\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:12.036715 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.036500    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:12.086567 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.086504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:12.235706 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.235664    2563 generic.go:356] "Generic (PLEG): container finished" podID="29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" containerID="a62849749085bdcab01244f9f922fa713b2b4954bfb15d569fa7114e8f3ce1b8" exitCode=0
Sep 09 20:35:12.235882 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.235714    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"
Sep 09 20:35:12.235882 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.235728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6" event={"ID":"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd","Type":"ContainerDied","Data":"a62849749085bdcab01244f9f922fa713b2b4954bfb15d569fa7114e8f3ce1b8"}
Sep 09 20:35:12.235882 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.235787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6" event={"ID":"29ce2c22-df27-47db-a0ca-0e22ae1ca5dd","Type":"ContainerDied","Data":"aa4c3ea2d7f1785c65e7181bd803b51843a40df52393a573903d869d5ca0c7b7"}
Sep 09 20:35:12.235882 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.235809    2563 scope.go:117] "RemoveContainer" containerID="a62849749085bdcab01244f9f922fa713b2b4954bfb15d569fa7114e8f3ce1b8"
Sep 09 20:35:12.244963 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.244938    2563 scope.go:117] "RemoveContainer" containerID="a62849749085bdcab01244f9f922fa713b2b4954bfb15d569fa7114e8f3ce1b8"
Sep 09 20:35:12.245255 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:35:12.245235    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62849749085bdcab01244f9f922fa713b2b4954bfb15d569fa7114e8f3ce1b8\": container with ID starting with a62849749085bdcab01244f9f922fa713b2b4954bfb15d569fa7114e8f3ce1b8 not found: ID does not exist" containerID="a62849749085bdcab01244f9f922fa713b2b4954bfb15d569fa7114e8f3ce1b8"
Sep 09 20:35:12.245302 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.245265    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62849749085bdcab01244f9f922fa713b2b4954bfb15d569fa7114e8f3ce1b8"} err="failed to get container status \"a62849749085bdcab01244f9f922fa713b2b4954bfb15d569fa7114e8f3ce1b8\": rpc error: code = NotFound desc = could not find container \"a62849749085bdcab01244f9f922fa713b2b4954bfb15d569fa7114e8f3ce1b8\": container with ID starting with a62849749085bdcab01244f9f922fa713b2b4954bfb15d569fa7114e8f3ce1b8 not found: ID does not exist"
Sep 09 20:35:12.262349 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.262316    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"]
Sep 09 20:35:12.271491 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:12.271458    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-82f543-openshift-default-746d47c544-g7vn6"]
Sep 09 20:35:13.319063 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:13.319028    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" path="/var/lib/kubelet/pods/29ce2c22-df27-47db-a0ca-0e22ae1ca5dd/volumes"
Sep 09 20:35:14.385393 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.385365    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:35:14.457246 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457217    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-token\") pod \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") "
Sep 09 20:35:14.457408 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457257    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-data\") pod \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") "
Sep 09 20:35:14.457408 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457286    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-workload-socket\") pod \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") "
Sep 09 20:35:14.457408 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457341    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-envoy\") pod \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") "
Sep 09 20:35:14.457408 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457379    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istiod-ca-cert\") pod \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") "
Sep 09 20:35:14.457628 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457438    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-workload-certs\") pod \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") "
Sep 09 20:35:14.457628 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457467    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-credential-socket\") pod \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") "
Sep 09 20:35:14.457628 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457499    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g75hd\" (UniqueName: \"kubernetes.io/projected/5246aabf-1ade-4cb6-b509-88886a1a4ecd-kube-api-access-g75hd\") pod \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") "
Sep 09 20:35:14.457628 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457525    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-podinfo\") pod \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\" (UID: \"5246aabf-1ade-4cb6-b509-88886a1a4ecd\") "
Sep 09 20:35:14.457628 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457526    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-data" (OuterVolumeSpecName: "istio-data") pod "5246aabf-1ade-4cb6-b509-88886a1a4ecd" (UID: "5246aabf-1ade-4cb6-b509-88886a1a4ecd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:14.457901 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457736    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5246aabf-1ade-4cb6-b509-88886a1a4ecd" (UID: "5246aabf-1ade-4cb6-b509-88886a1a4ecd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:14.457901 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457751    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5246aabf-1ade-4cb6-b509-88886a1a4ecd" (UID: "5246aabf-1ade-4cb6-b509-88886a1a4ecd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:14.457901 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457835    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5246aabf-1ade-4cb6-b509-88886a1a4ecd" (UID: "5246aabf-1ade-4cb6-b509-88886a1a4ecd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:14.458053 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.457951    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5246aabf-1ade-4cb6-b509-88886a1a4ecd" (UID: "5246aabf-1ade-4cb6-b509-88886a1a4ecd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:14.458053 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.458017    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:14.458053 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.458033    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:14.458053 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.458044    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:14.458053 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.458054    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:14.459618 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.459560    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-token" (OuterVolumeSpecName: "istio-token") pod "5246aabf-1ade-4cb6-b509-88886a1a4ecd" (UID: "5246aabf-1ade-4cb6-b509-88886a1a4ecd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:14.459737 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.459668    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5246aabf-1ade-4cb6-b509-88886a1a4ecd" (UID: "5246aabf-1ade-4cb6-b509-88886a1a4ecd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:14.459737 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.459707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5246aabf-1ade-4cb6-b509-88886a1a4ecd" (UID: "5246aabf-1ade-4cb6-b509-88886a1a4ecd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:35:14.459737 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.459727    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5246aabf-1ade-4cb6-b509-88886a1a4ecd-kube-api-access-g75hd" (OuterVolumeSpecName: "kube-api-access-g75hd") pod "5246aabf-1ade-4cb6-b509-88886a1a4ecd" (UID: "5246aabf-1ade-4cb6-b509-88886a1a4ecd"). InnerVolumeSpecName "kube-api-access-g75hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:14.558932 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.558887    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:14.558932 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.558925    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g75hd\" (UniqueName: \"kubernetes.io/projected/5246aabf-1ade-4cb6-b509-88886a1a4ecd-kube-api-access-g75hd\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:14.558932 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.558936    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:14.558932 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.558945    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5246aabf-1ade-4cb6-b509-88886a1a4ecd-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:14.559211 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:14.558953    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246aabf-1ade-4cb6-b509-88886a1a4ecd-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:15.253475 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:15.253440    2563 generic.go:356] "Generic (PLEG): container finished" podID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerID="d321909f1d05c52db1cfbdb1223238e04e3fefd767ada8caf9bd68a2bb6af580" exitCode=0
Sep 09 20:35:15.253673 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:15.253492    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"
Sep 09 20:35:15.253673 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:15.253523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" event={"ID":"5246aabf-1ade-4cb6-b509-88886a1a4ecd","Type":"ContainerDied","Data":"d321909f1d05c52db1cfbdb1223238e04e3fefd767ada8caf9bd68a2bb6af580"}
Sep 09 20:35:15.253673 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:15.253603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg" event={"ID":"5246aabf-1ade-4cb6-b509-88886a1a4ecd","Type":"ContainerDied","Data":"303cc4d671bd87eb97fef41474838cb6dbe70a2089183e988dc5ae7be6021cf0"}
Sep 09 20:35:15.253673 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:15.253637    2563 scope.go:117] "RemoveContainer" containerID="d321909f1d05c52db1cfbdb1223238e04e3fefd767ada8caf9bd68a2bb6af580"
Sep 09 20:35:15.262695 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:15.262674    2563 scope.go:117] "RemoveContainer" containerID="5349a80693b196631c859861b80287b52bc52a3613be3f91217b3dcb0249206f"
Sep 09 20:35:15.271201 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:15.271176    2563 scope.go:117] "RemoveContainer" containerID="d321909f1d05c52db1cfbdb1223238e04e3fefd767ada8caf9bd68a2bb6af580"
Sep 09 20:35:15.271491 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:35:15.271472    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d321909f1d05c52db1cfbdb1223238e04e3fefd767ada8caf9bd68a2bb6af580\": container with ID starting with d321909f1d05c52db1cfbdb1223238e04e3fefd767ada8caf9bd68a2bb6af580 not found: ID does not exist" containerID="d321909f1d05c52db1cfbdb1223238e04e3fefd767ada8caf9bd68a2bb6af580"
Sep 09 20:35:15.271544 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:15.271502    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d321909f1d05c52db1cfbdb1223238e04e3fefd767ada8caf9bd68a2bb6af580"} err="failed to get container status \"d321909f1d05c52db1cfbdb1223238e04e3fefd767ada8caf9bd68a2bb6af580\": rpc error: code = NotFound desc = could not find container \"d321909f1d05c52db1cfbdb1223238e04e3fefd767ada8caf9bd68a2bb6af580\": container with ID starting with d321909f1d05c52db1cfbdb1223238e04e3fefd767ada8caf9bd68a2bb6af580 not found: ID does not exist"
Sep 09 20:35:15.271544 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:15.271519    2563 scope.go:117] "RemoveContainer" containerID="5349a80693b196631c859861b80287b52bc52a3613be3f91217b3dcb0249206f"
Sep 09 20:35:15.271814 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:35:15.271789    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5349a80693b196631c859861b80287b52bc52a3613be3f91217b3dcb0249206f\": container with ID starting with 5349a80693b196631c859861b80287b52bc52a3613be3f91217b3dcb0249206f not found: ID does not exist" containerID="5349a80693b196631c859861b80287b52bc52a3613be3f91217b3dcb0249206f"
Sep 09 20:35:15.271856 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:15.271825    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5349a80693b196631c859861b80287b52bc52a3613be3f91217b3dcb0249206f"} err="failed to get container status \"5349a80693b196631c859861b80287b52bc52a3613be3f91217b3dcb0249206f\": rpc error: code = NotFound desc = could not find container \"5349a80693b196631c859861b80287b52bc52a3613be3f91217b3dcb0249206f\": container with ID starting with 5349a80693b196631c859861b80287b52bc52a3613be3f91217b3dcb0249206f not found: ID does not exist"
Sep 09 20:35:15.276815 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:15.276785    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"]
Sep 09 20:35:15.287922 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:15.287892    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-36f068b3-openshift-default-74c4bd4568-ktkvg"]
Sep 09 20:35:15.312910 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:15.312877    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" path="/var/lib/kubelet/pods/5246aabf-1ade-4cb6-b509-88886a1a4ecd/volumes"
Sep 09 20:35:16.079248 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:16.079217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:16.183254 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:16.183216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:16.313365 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:16.313328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:16.401311 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:16.401228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:16.519123 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:16.519088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:16.878808 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:16.878772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:17.884009 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:17.883975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:17.998584 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:17.998547    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk"]
Sep 09 20:35:17.998890 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:17.998877    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" containerName="istio-proxy"
Sep 09 20:35:17.998890 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:17.998890    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" containerName="istio-proxy"
Sep 09 20:35:17.998978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:17.998908    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0154e5b-130e-47a2-8545-b4a252955a21" containerName="payload-processing"
Sep 09 20:35:17.998978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:17.998913    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0154e5b-130e-47a2-8545-b4a252955a21" containerName="payload-processing"
Sep 09 20:35:17.998978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:17.998921    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerName="istio-proxy"
Sep 09 20:35:17.998978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:17.998927    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerName="istio-proxy"
Sep 09 20:35:17.998978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:17.998937    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerName="istio-proxy"
Sep 09 20:35:17.998978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:17.998942    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerName="istio-proxy"
Sep 09 20:35:17.999153 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:17.998989    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0154e5b-130e-47a2-8545-b4a252955a21" containerName="payload-processing"
Sep 09 20:35:17.999153 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:17.998996    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerName="istio-proxy"
Sep 09 20:35:17.999153 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:17.999004    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="29ce2c22-df27-47db-a0ca-0e22ae1ca5dd" containerName="istio-proxy"
Sep 09 20:35:18.003280 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:18.003257    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk"
Sep 09 20:35:18.005541 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:18.005520    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:35:18.005683 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:18.005541    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:35:18.010892 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:18.010510    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk"]
Sep 09 20:35:18.090865 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:18.090822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fd4875-fbbd-4cc0-91f4-1f1ede0631e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk\" (UID: \"41fd4875-fbbd-4cc0-91f4-1f1ede0631e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk"
Sep 09 20:35:18.191894 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:18.191803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fd4875-fbbd-4cc0-91f4-1f1ede0631e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk\" (UID: \"41fd4875-fbbd-4cc0-91f4-1f1ede0631e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk"
Sep 09 20:35:18.192452 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:18.192426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fd4875-fbbd-4cc0-91f4-1f1ede0631e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk\" (UID: \"41fd4875-fbbd-4cc0-91f4-1f1ede0631e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk"
Sep 09 20:35:18.244170 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:18.244126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:18.315174 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:18.315137    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk"
Sep 09 20:35:18.588900 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:18.588876    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk"]
Sep 09 20:35:18.592624 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:35:18.592590    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd4875_fbbd_4cc0_91f4_1f1ede0631e0.slice/crio-dc2ffe94b5d7c07a5770f635e2f924e209d1f018ddbe860247c190a454845b2a WatchSource:0}: Error finding container dc2ffe94b5d7c07a5770f635e2f924e209d1f018ddbe860247c190a454845b2a: Status 404 returned error can't find the container with id dc2ffe94b5d7c07a5770f635e2f924e209d1f018ddbe860247c190a454845b2a
Sep 09 20:35:19.014627 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:19.014563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:19.275834 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:19.275799    2563 generic.go:356] "Generic (PLEG): container finished" podID="41fd4875-fbbd-4cc0-91f4-1f1ede0631e0" containerID="200ec5154ec8ff4d47850ebd044289faee9f82cefd8a9b6ed238f0a95805a77a" exitCode=0
Sep 09 20:35:19.276034 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:19.275880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk" event={"ID":"41fd4875-fbbd-4cc0-91f4-1f1ede0631e0","Type":"ContainerDied","Data":"200ec5154ec8ff4d47850ebd044289faee9f82cefd8a9b6ed238f0a95805a77a"}
Sep 09 20:35:19.276034 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:19.275959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk" event={"ID":"41fd4875-fbbd-4cc0-91f4-1f1ede0631e0","Type":"ContainerStarted","Data":"dc2ffe94b5d7c07a5770f635e2f924e209d1f018ddbe860247c190a454845b2a"}
Sep 09 20:35:19.807337 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:19.807291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:20.409411 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:20.409386    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk"
Sep 09 20:35:20.495388 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:20.495355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:20.515326 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:20.515289    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fd4875-fbbd-4cc0-91f4-1f1ede0631e0-maas-api-service-ca\") pod \"41fd4875-fbbd-4cc0-91f4-1f1ede0631e0\" (UID: \"41fd4875-fbbd-4cc0-91f4-1f1ede0631e0\") "
Sep 09 20:35:20.515686 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:20.515663    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41fd4875-fbbd-4cc0-91f4-1f1ede0631e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41fd4875-fbbd-4cc0-91f4-1f1ede0631e0" (UID: "41fd4875-fbbd-4cc0-91f4-1f1ede0631e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:20.616054 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:20.615964    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fd4875-fbbd-4cc0-91f4-1f1ede0631e0-maas-api-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:21.206328 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:21.206291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:35:21.286052 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:21.286027    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk"
Sep 09 20:35:21.286254 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:21.286024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk" event={"ID":"41fd4875-fbbd-4cc0-91f4-1f1ede0631e0","Type":"ContainerDied","Data":"dc2ffe94b5d7c07a5770f635e2f924e209d1f018ddbe860247c190a454845b2a"}
Sep 09 20:35:21.286254 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:21.286152    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc2ffe94b5d7c07a5770f635e2f924e209d1f018ddbe860247c190a454845b2a"
Sep 09 20:35:22.616230 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:22.616201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:22.718764 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:22.718728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:23.716317 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:23.716280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:24.018447 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:24.018349    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:24.507858 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:24.507826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:26.783560 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:26.783515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:27.366534 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:27.366493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:27.463088 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:27.463056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:27.563619 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:27.563568    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:28.479314 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:28.479266    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:29.251461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.251426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:29.352486 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.352447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:29.755402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.755359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:29.859892 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.859855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:29.907099 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.907066    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"]
Sep 09 20:35:29.907403 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.907390    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41fd4875-fbbd-4cc0-91f4-1f1ede0631e0" containerName="revoke-keys"
Sep 09 20:35:29.907448 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.907405    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="41fd4875-fbbd-4cc0-91f4-1f1ede0631e0" containerName="revoke-keys"
Sep 09 20:35:29.907491 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.907480    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="41fd4875-fbbd-4cc0-91f4-1f1ede0631e0" containerName="revoke-keys"
Sep 09 20:35:29.907524 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.907492    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5246aabf-1ade-4cb6-b509-88886a1a4ecd" containerName="istio-proxy"
Sep 09 20:35:29.910503 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.910486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"
Sep 09 20:35:29.914781 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.914759    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-dc2ebb\""
Sep 09 20:35:29.929934 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.929904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"]
Sep 09 20:35:29.971540 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.971504    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"]
Sep 09 20:35:29.975229 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.975201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"
Sep 09 20:35:29.978527 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.978506    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-dc2ebb-dockercfg-6r8wr\""
Sep 09 20:35:29.993727 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.993702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"]
Sep 09 20:35:29.994874 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:29.994818    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:30.002644 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.002613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9849330-7ffc-4835-bfd6-feec46672d6c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft\" (UID: \"c9849330-7ffc-4835-bfd6-feec46672d6c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"
Sep 09 20:35:30.002759 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.002658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw77q\" (UniqueName: \"kubernetes.io/projected/c9849330-7ffc-4835-bfd6-feec46672d6c-kube-api-access-nw77q\") pod \"payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft\" (UID: \"c9849330-7ffc-4835-bfd6-feec46672d6c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"
Sep 09 20:35:30.104237 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.104140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwngs\" (UniqueName: \"kubernetes.io/projected/89a749bb-51b5-4a69-b3ef-9fba029fce33-kube-api-access-vwngs\") pod \"payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw\" (UID: \"89a749bb-51b5-4a69-b3ef-9fba029fce33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"
Sep 09 20:35:30.104395 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.104284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a749bb-51b5-4a69-b3ef-9fba029fce33-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw\" (UID: \"89a749bb-51b5-4a69-b3ef-9fba029fce33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"
Sep 09 20:35:30.104395 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.104325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9849330-7ffc-4835-bfd6-feec46672d6c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft\" (UID: \"c9849330-7ffc-4835-bfd6-feec46672d6c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"
Sep 09 20:35:30.104395 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.104343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw77q\" (UniqueName: \"kubernetes.io/projected/c9849330-7ffc-4835-bfd6-feec46672d6c-kube-api-access-nw77q\") pod \"payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft\" (UID: \"c9849330-7ffc-4835-bfd6-feec46672d6c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"
Sep 09 20:35:30.105017 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.104996    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9849330-7ffc-4835-bfd6-feec46672d6c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft\" (UID: \"c9849330-7ffc-4835-bfd6-feec46672d6c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"
Sep 09 20:35:30.112241 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.112207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw77q\" (UniqueName: \"kubernetes.io/projected/c9849330-7ffc-4835-bfd6-feec46672d6c-kube-api-access-nw77q\") pod \"payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft\" (UID: \"c9849330-7ffc-4835-bfd6-feec46672d6c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"
Sep 09 20:35:30.205078 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.205041    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a749bb-51b5-4a69-b3ef-9fba029fce33-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw\" (UID: \"89a749bb-51b5-4a69-b3ef-9fba029fce33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"
Sep 09 20:35:30.205239 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.205098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwngs\" (UniqueName: \"kubernetes.io/projected/89a749bb-51b5-4a69-b3ef-9fba029fce33-kube-api-access-vwngs\") pod \"payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw\" (UID: \"89a749bb-51b5-4a69-b3ef-9fba029fce33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"
Sep 09 20:35:30.205812 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.205791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a749bb-51b5-4a69-b3ef-9fba029fce33-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw\" (UID: \"89a749bb-51b5-4a69-b3ef-9fba029fce33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"
Sep 09 20:35:30.213004 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.212978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwngs\" (UniqueName: \"kubernetes.io/projected/89a749bb-51b5-4a69-b3ef-9fba029fce33-kube-api-access-vwngs\") pod \"payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw\" (UID: \"89a749bb-51b5-4a69-b3ef-9fba029fce33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"
Sep 09 20:35:30.220846 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.220823    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"
Sep 09 20:35:30.275608 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.275559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:30.286156 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.285890    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"
Sep 09 20:35:30.496793 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.496759    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"]
Sep 09 20:35:30.500330 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:35:30.500304    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9849330_7ffc_4835_bfd6_feec46672d6c.slice/crio-2803eb05b67d4ada89b5f0b7425fe6129f65d8b33589994114b4ff852535464b WatchSource:0}: Error finding container 2803eb05b67d4ada89b5f0b7425fe6129f65d8b33589994114b4ff852535464b: Status 404 returned error can't find the container with id 2803eb05b67d4ada89b5f0b7425fe6129f65d8b33589994114b4ff852535464b
Sep 09 20:35:30.563907 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.563880    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"]
Sep 09 20:35:30.565660 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:35:30.565628    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a749bb_51b5_4a69_b3ef_9fba029fce33.slice/crio-bf739125a3bab876e42272a34b9b4c94eb2ca2640c04a18bf1b2cdd8f9035067 WatchSource:0}: Error finding container bf739125a3bab876e42272a34b9b4c94eb2ca2640c04a18bf1b2cdd8f9035067: Status 404 returned error can't find the container with id bf739125a3bab876e42272a34b9b4c94eb2ca2640c04a18bf1b2cdd8f9035067
Sep 09 20:35:30.972904 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:30.972868    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:31.064493 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.064446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:31.185382 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.185339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:31.252852 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.252768    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk"]
Sep 09 20:35:31.258422 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.258393    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-21117b-sq5bk"]
Sep 09 20:35:31.260361 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.260318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:31.313227 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.313196    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41fd4875-fbbd-4cc0-91f4-1f1ede0631e0" path="/var/lib/kubelet/pods/41fd4875-fbbd-4cc0-91f4-1f1ede0631e0/volumes"
Sep 09 20:35:31.337084 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.337048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw" event={"ID":"89a749bb-51b5-4a69-b3ef-9fba029fce33","Type":"ContainerStarted","Data":"01a633b54401552e2cbb8fd889958c30e43143645d937babe4b754758e678ace"}
Sep 09 20:35:31.337272 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.337108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw" event={"ID":"89a749bb-51b5-4a69-b3ef-9fba029fce33","Type":"ContainerStarted","Data":"bf739125a3bab876e42272a34b9b4c94eb2ca2640c04a18bf1b2cdd8f9035067"}
Sep 09 20:35:31.337272 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.337155    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"
Sep 09 20:35:31.338801 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.338775    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft" event={"ID":"c9849330-7ffc-4835-bfd6-feec46672d6c","Type":"ContainerStarted","Data":"cbc1a244c63c12a9486bce27a1f9a71436d1e4caa2a711f63398304256248c48"}
Sep 09 20:35:31.338908 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.338818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft" event={"ID":"c9849330-7ffc-4835-bfd6-feec46672d6c","Type":"ContainerStarted","Data":"2803eb05b67d4ada89b5f0b7425fe6129f65d8b33589994114b4ff852535464b"}
Sep 09 20:35:31.338954 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.338918    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"
Sep 09 20:35:31.364142 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.364095    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw" podStartSLOduration=2.364081139 podStartE2EDuration="2.364081139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:31.362802235 +0000 UTC m=+1138.601904537" watchObservedRunningTime="2026-09-09 20:35:31.364081139 +0000 UTC m=+1138.603183438"
Sep 09 20:35:31.365675 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.365647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:31.403140 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.403079    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft" podStartSLOduration=2.403060902 podStartE2EDuration="2.403060902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:31.400810525 +0000 UTC m=+1138.639912825" watchObservedRunningTime="2026-09-09 20:35:31.403060902 +0000 UTC m=+1138.642163201"
Sep 09 20:35:31.716155 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:31.716120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:32.236901 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:32.236867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:33.192161 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:33.192114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:33.984048 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:33.984011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:34.509951 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:34.509913    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:35:35.109710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:35.109672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:36.769906 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:36.769847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:37.742312 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:37.742271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:37.845663 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:37.845618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:38.010878 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:38.010787    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"]
Sep 09 20:35:38.011051 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:38.011025    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy" containerID="cri-o://4ddff32e89e1316fa334888e3ac654c67a84a974cd5366d7aa46d586927f5a1e" gracePeriod=30
Sep 09 20:35:38.042240 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:38.042202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:38.697441 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:38.697399    2563 patch_prober.go:28] interesting pod/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 20:35:38.697634 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:38.697470    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:38.905564 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:38.905526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:39.901048 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:39.901015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:40.692626 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:40.692556    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:41.559380 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:41.559343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:42.346504 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:42.346472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"
Sep 09 20:35:42.347011 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:42.346710    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"
Sep 09 20:35:42.372049 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:42.372013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:42.505852 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:42.505812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:42.722196 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:42.722153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:43.152507 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.152461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:43.286551 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.286482    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:35:43.347000 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.346961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:43.400060 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.400020    2563 generic.go:356] "Generic (PLEG): container finished" podID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerID="4ddff32e89e1316fa334888e3ac654c67a84a974cd5366d7aa46d586927f5a1e" exitCode=0
Sep 09 20:35:43.400292 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.400077    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"
Sep 09 20:35:43.400292 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.400089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" event={"ID":"1314a2b3-c135-4998-99b5-76a9b81370c2","Type":"ContainerDied","Data":"4ddff32e89e1316fa334888e3ac654c67a84a974cd5366d7aa46d586927f5a1e"}
Sep 09 20:35:43.400292 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.400140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh" event={"ID":"1314a2b3-c135-4998-99b5-76a9b81370c2","Type":"ContainerDied","Data":"27b9943c3f8c177a039f06e8dc62764c84411a31f1d8e25ff46271aa7f316232"}
Sep 09 20:35:43.400292 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.400166    2563 scope.go:117] "RemoveContainer" containerID="4ddff32e89e1316fa334888e3ac654c67a84a974cd5366d7aa46d586927f5a1e"
Sep 09 20:35:43.408873 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.408854    2563 scope.go:117] "RemoveContainer" containerID="b798fa556afa4f3de4628b0a36fa674b416b896bedb60cb7fd71569bf37fffce"
Sep 09 20:35:43.418462 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.418441    2563 scope.go:117] "RemoveContainer" containerID="4ddff32e89e1316fa334888e3ac654c67a84a974cd5366d7aa46d586927f5a1e"
Sep 09 20:35:43.418807 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:35:43.418784    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ddff32e89e1316fa334888e3ac654c67a84a974cd5366d7aa46d586927f5a1e\": container with ID starting with 4ddff32e89e1316fa334888e3ac654c67a84a974cd5366d7aa46d586927f5a1e not found: ID does not exist" containerID="4ddff32e89e1316fa334888e3ac654c67a84a974cd5366d7aa46d586927f5a1e"
Sep 09 20:35:43.418926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.418815    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddff32e89e1316fa334888e3ac654c67a84a974cd5366d7aa46d586927f5a1e"} err="failed to get container status \"4ddff32e89e1316fa334888e3ac654c67a84a974cd5366d7aa46d586927f5a1e\": rpc error: code = NotFound desc = could not find container \"4ddff32e89e1316fa334888e3ac654c67a84a974cd5366d7aa46d586927f5a1e\": container with ID starting with 4ddff32e89e1316fa334888e3ac654c67a84a974cd5366d7aa46d586927f5a1e not found: ID does not exist"
Sep 09 20:35:43.418926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.418842    2563 scope.go:117] "RemoveContainer" containerID="b798fa556afa4f3de4628b0a36fa674b416b896bedb60cb7fd71569bf37fffce"
Sep 09 20:35:43.419137 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:35:43.419118    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b798fa556afa4f3de4628b0a36fa674b416b896bedb60cb7fd71569bf37fffce\": container with ID starting with b798fa556afa4f3de4628b0a36fa674b416b896bedb60cb7fd71569bf37fffce not found: ID does not exist" containerID="b798fa556afa4f3de4628b0a36fa674b416b896bedb60cb7fd71569bf37fffce"
Sep 09 20:35:43.419177 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.419144    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b798fa556afa4f3de4628b0a36fa674b416b896bedb60cb7fd71569bf37fffce"} err="failed to get container status \"b798fa556afa4f3de4628b0a36fa674b416b896bedb60cb7fd71569bf37fffce\": rpc error: code = NotFound desc = could not find container \"b798fa556afa4f3de4628b0a36fa674b416b896bedb60cb7fd71569bf37fffce\": container with ID starting with b798fa556afa4f3de4628b0a36fa674b416b896bedb60cb7fd71569bf37fffce not found: ID does not exist"
Sep 09 20:35:43.430596 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.430557    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-credential-socket\") pod \"1314a2b3-c135-4998-99b5-76a9b81370c2\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") "
Sep 09 20:35:43.430732 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.430605    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-envoy\") pod \"1314a2b3-c135-4998-99b5-76a9b81370c2\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") "
Sep 09 20:35:43.430732 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.430634    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1314a2b3-c135-4998-99b5-76a9b81370c2-istiod-ca-cert\") pod \"1314a2b3-c135-4998-99b5-76a9b81370c2\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") "
Sep 09 20:35:43.430732 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.430654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-workload-socket\") pod \"1314a2b3-c135-4998-99b5-76a9b81370c2\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") "
Sep 09 20:35:43.430732 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.430718    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-podinfo\") pod \"1314a2b3-c135-4998-99b5-76a9b81370c2\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") "
Sep 09 20:35:43.430941 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.430756    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqdmq\" (UniqueName: \"kubernetes.io/projected/1314a2b3-c135-4998-99b5-76a9b81370c2-kube-api-access-jqdmq\") pod \"1314a2b3-c135-4998-99b5-76a9b81370c2\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") "
Sep 09 20:35:43.430941 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.430783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-workload-certs\") pod \"1314a2b3-c135-4998-99b5-76a9b81370c2\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") "
Sep 09 20:35:43.430941 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.430840    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1314a2b3-c135-4998-99b5-76a9b81370c2" (UID: "1314a2b3-c135-4998-99b5-76a9b81370c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:43.430941 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.430856    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-data\") pod \"1314a2b3-c135-4998-99b5-76a9b81370c2\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") "
Sep 09 20:35:43.430941 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.430881    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-token\") pod \"1314a2b3-c135-4998-99b5-76a9b81370c2\" (UID: \"1314a2b3-c135-4998-99b5-76a9b81370c2\") "
Sep 09 20:35:43.431210 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.431044    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1314a2b3-c135-4998-99b5-76a9b81370c2" (UID: "1314a2b3-c135-4998-99b5-76a9b81370c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:43.431210 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.431077    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1314a2b3-c135-4998-99b5-76a9b81370c2" (UID: "1314a2b3-c135-4998-99b5-76a9b81370c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:43.431210 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.431090    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-data" (OuterVolumeSpecName: "istio-data") pod "1314a2b3-c135-4998-99b5-76a9b81370c2" (UID: "1314a2b3-c135-4998-99b5-76a9b81370c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:43.431210 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.431155    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1314a2b3-c135-4998-99b5-76a9b81370c2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1314a2b3-c135-4998-99b5-76a9b81370c2" (UID: "1314a2b3-c135-4998-99b5-76a9b81370c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:43.431210 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.431192    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:43.431210 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.431211    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:43.431476 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.431225    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:43.431476 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.431239    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:43.432929 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.432889    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1314a2b3-c135-4998-99b5-76a9b81370c2" (UID: "1314a2b3-c135-4998-99b5-76a9b81370c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:43.433008 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.432963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1314a2b3-c135-4998-99b5-76a9b81370c2" (UID: "1314a2b3-c135-4998-99b5-76a9b81370c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:35:43.433008 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.432986    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-token" (OuterVolumeSpecName: "istio-token") pod "1314a2b3-c135-4998-99b5-76a9b81370c2" (UID: "1314a2b3-c135-4998-99b5-76a9b81370c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:43.433129 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.433109    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1314a2b3-c135-4998-99b5-76a9b81370c2-kube-api-access-jqdmq" (OuterVolumeSpecName: "kube-api-access-jqdmq") pod "1314a2b3-c135-4998-99b5-76a9b81370c2" (UID: "1314a2b3-c135-4998-99b5-76a9b81370c2"). InnerVolumeSpecName "kube-api-access-jqdmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:43.520321 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.520283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:43.531563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.531533    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:43.531563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.531560    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqdmq\" (UniqueName: \"kubernetes.io/projected/1314a2b3-c135-4998-99b5-76a9b81370c2-kube-api-access-jqdmq\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:43.531769 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.531589    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:43.531769 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.531602    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1314a2b3-c135-4998-99b5-76a9b81370c2-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:43.531769 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.531609    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1314a2b3-c135-4998-99b5-76a9b81370c2-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:43.733377 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.733321    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"]
Sep 09 20:35:43.735374 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:43.735346    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-21117b-openshift-default-6778ddc5fd-ph5lh"]
Sep 09 20:35:44.241145 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:44.241113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:45.313485 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:45.313447    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" path="/var/lib/kubelet/pods/1314a2b3-c135-4998-99b5-76a9b81370c2/volumes"
Sep 09 20:35:45.924737 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:45.924699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:46.524632 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:46.524593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:46.608223 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:46.608185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:47.007074 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:47.007037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:48.091022 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:48.090987    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:48.997761 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:48.997726    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"]
Sep 09 20:35:48.998107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:48.998095    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy"
Sep 09 20:35:48.998148 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:48.998108    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy"
Sep 09 20:35:48.998183 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:48.998163    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy"
Sep 09 20:35:48.998183 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:48.998172    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy"
Sep 09 20:35:49.000143 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.000110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:49.002438 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.002416    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.004802 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.004778    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-47e11e-serving-cert\""
Sep 09 20:35:49.004985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.004837    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:35:49.004985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.004848    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bbx7f\""
Sep 09 20:35:49.004985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.004849    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 20:35:49.005823 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.005806    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 20:35:49.013396 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.013375    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"]
Sep 09 20:35:49.066985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.066944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:49.076180 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.076132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.076376 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.076238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.076376 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.076295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ckpk\" (UniqueName: \"kubernetes.io/projected/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-kube-api-access-4ckpk\") pod \"maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.076376 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.076341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.174628 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.174554    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:49.177626 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.177598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.177753 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.177647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ckpk\" (UniqueName: \"kubernetes.io/projected/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-kube-api-access-4ckpk\") pod \"maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.177753 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.177664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.177753 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.177708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.178431 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.178408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.180072 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.180053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.180137 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.180102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.187710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.187682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ckpk\" (UniqueName: \"kubernetes.io/projected/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-kube-api-access-4ckpk\") pod \"maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.314376 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.314287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:49.493412 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.493372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:49.590693 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.590607    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"]
Sep 09 20:35:49.593657 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:35:49.593622    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce4d80e_a5b2_456e_acd8_772c3bf9dd09.slice/crio-1f6643431befaf440af74d6b0ef1a3085d6fe4274dbafd829dee52bab6998b1e WatchSource:0}: Error finding container 1f6643431befaf440af74d6b0ef1a3085d6fe4274dbafd829dee52bab6998b1e: Status 404 returned error can't find the container with id 1f6643431befaf440af74d6b0ef1a3085d6fe4274dbafd829dee52bab6998b1e
Sep 09 20:35:49.594110 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.594077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:49.595074 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.595058    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:35:49.841611 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:49.841490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:50.075931 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:50.075889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:50.439548 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:50.439514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs" event={"ID":"dce4d80e-a5b2-456e-acd8-772c3bf9dd09","Type":"ContainerStarted","Data":"1f6643431befaf440af74d6b0ef1a3085d6fe4274dbafd829dee52bab6998b1e"}
Sep 09 20:35:51.000831 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:51.000795    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:51.646226 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:51.646182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:51.671488 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:51.670800    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"]
Sep 09 20:35:51.671488 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:51.671060    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5" podUID="0ddea5f1-7cb2-4df5-ae87-1f900be04a67" containerName="payload-pre-processing" containerID="cri-o://5ebd9d5e04d0d2bb2b80f241fe29767d26cd8d4a868ac585bc402f6d2f719b00" gracePeriod=30
Sep 09 20:35:51.747057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:51.747014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:51.821507 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:51.821459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:51.876653 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:51.876605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:52.043297 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.043231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:52.266638 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.266604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:52.406795 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.406766    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"
Sep 09 20:35:52.453422 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.453389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs" event={"ID":"dce4d80e-a5b2-456e-acd8-772c3bf9dd09","Type":"ContainerStarted","Data":"a6466a02222df3df12cf2b2253e1a680f8fb836fc2cf15598e877907e3af0f39"}
Sep 09 20:35:52.453655 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.453510    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:52.454899 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.454873    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ddea5f1-7cb2-4df5-ae87-1f900be04a67" containerID="5ebd9d5e04d0d2bb2b80f241fe29767d26cd8d4a868ac585bc402f6d2f719b00" exitCode=0
Sep 09 20:35:52.454899 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.454894    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"
Sep 09 20:35:52.455045 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.454936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5" event={"ID":"0ddea5f1-7cb2-4df5-ae87-1f900be04a67","Type":"ContainerDied","Data":"5ebd9d5e04d0d2bb2b80f241fe29767d26cd8d4a868ac585bc402f6d2f719b00"}
Sep 09 20:35:52.455045 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.454994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5" event={"ID":"0ddea5f1-7cb2-4df5-ae87-1f900be04a67","Type":"ContainerDied","Data":"59c6a9f5097bddce6f919a0605cf30e650765a42651c9c45a7e553bcc95561e3"}
Sep 09 20:35:52.455045 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.455014    2563 scope.go:117] "RemoveContainer" containerID="5ebd9d5e04d0d2bb2b80f241fe29767d26cd8d4a868ac585bc402f6d2f719b00"
Sep 09 20:35:52.464710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.464690    2563 scope.go:117] "RemoveContainer" containerID="5ebd9d5e04d0d2bb2b80f241fe29767d26cd8d4a868ac585bc402f6d2f719b00"
Sep 09 20:35:52.465036 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:35:52.465018    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebd9d5e04d0d2bb2b80f241fe29767d26cd8d4a868ac585bc402f6d2f719b00\": container with ID starting with 5ebd9d5e04d0d2bb2b80f241fe29767d26cd8d4a868ac585bc402f6d2f719b00 not found: ID does not exist" containerID="5ebd9d5e04d0d2bb2b80f241fe29767d26cd8d4a868ac585bc402f6d2f719b00"
Sep 09 20:35:52.465098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.465046    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebd9d5e04d0d2bb2b80f241fe29767d26cd8d4a868ac585bc402f6d2f719b00"} err="failed to get container status \"5ebd9d5e04d0d2bb2b80f241fe29767d26cd8d4a868ac585bc402f6d2f719b00\": rpc error: code = NotFound desc = could not find container \"5ebd9d5e04d0d2bb2b80f241fe29767d26cd8d4a868ac585bc402f6d2f719b00\": container with ID starting with 5ebd9d5e04d0d2bb2b80f241fe29767d26cd8d4a868ac585bc402f6d2f719b00 not found: ID does not exist"
Sep 09 20:35:52.472326 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.472285    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs" podStartSLOduration=1.7619627599999999 podStartE2EDuration="4.472271834s" totalImagesPullingTime="2.710309074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:52.470433776 +0000 UTC m=+1159.709536079" watchObservedRunningTime="2026-09-09 20:35:52.472271834 +0000 UTC m=+1159.711374136"
Sep 09 20:35:52.507844 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.507809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2p78\" (UniqueName: \"kubernetes.io/projected/0ddea5f1-7cb2-4df5-ae87-1f900be04a67-kube-api-access-p2p78\") pod \"0ddea5f1-7cb2-4df5-ae87-1f900be04a67\" (UID: \"0ddea5f1-7cb2-4df5-ae87-1f900be04a67\") "
Sep 09 20:35:52.507998 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.507854    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ddea5f1-7cb2-4df5-ae87-1f900be04a67-plugins-config-volume\") pod \"0ddea5f1-7cb2-4df5-ae87-1f900be04a67\" (UID: \"0ddea5f1-7cb2-4df5-ae87-1f900be04a67\") "
Sep 09 20:35:52.508291 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.508262    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddea5f1-7cb2-4df5-ae87-1f900be04a67-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ddea5f1-7cb2-4df5-ae87-1f900be04a67" (UID: "0ddea5f1-7cb2-4df5-ae87-1f900be04a67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:52.509984 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.509964    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddea5f1-7cb2-4df5-ae87-1f900be04a67-kube-api-access-p2p78" (OuterVolumeSpecName: "kube-api-access-p2p78") pod "0ddea5f1-7cb2-4df5-ae87-1f900be04a67" (UID: "0ddea5f1-7cb2-4df5-ae87-1f900be04a67"). InnerVolumeSpecName "kube-api-access-p2p78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:52.608768 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.608686    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2p78\" (UniqueName: \"kubernetes.io/projected/0ddea5f1-7cb2-4df5-ae87-1f900be04a67-kube-api-access-p2p78\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:52.608768 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.608717    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ddea5f1-7cb2-4df5-ae87-1f900be04a67-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:35:52.742074 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.742040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:52.776400 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.776368    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"]
Sep 09 20:35:52.781068 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:52.781040    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4defd2-cd6d8df4-nvts5"]
Sep 09 20:35:53.313344 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:53.313308    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ddea5f1-7cb2-4df5-ae87-1f900be04a67" path="/var/lib/kubelet/pods/0ddea5f1-7cb2-4df5-ae87-1f900be04a67/volumes"
Sep 09 20:35:53.592677 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:53.592579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:54.215181 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:54.215145    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:35:54.369180 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:54.369137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:58.040985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:58.040902    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:58.141068 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:58.141032    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:58.390741 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:58.390661    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"]
Sep 09 20:35:58.390881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:58.390858    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy" containerID="cri-o://33ccb910b7e0611612eb40195633c732d071dae0e900edc6f3928efcdb5bbc18" gracePeriod=30
Sep 09 20:35:58.401497 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:58.401466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:58.465402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:58.465366    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:35:59.082226 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:59.082191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:59.253507 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:59.253465    2563 patch_prober.go:28] interesting pod/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 20:35:59.253678 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:59.253525    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:59.436394 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:59.436308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:59.507935 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:35:59.507890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:01.962943 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:01.962896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:02.687645 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:02.687606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:02.775989 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:02.775954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:03.508470 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.508439    2563 generic.go:356] "Generic (PLEG): container finished" podID="1f688849-f103-4589-8258-8e83a3015197" containerID="33ccb910b7e0611612eb40195633c732d071dae0e900edc6f3928efcdb5bbc18" exitCode=0
Sep 09 20:36:03.508833 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.508511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" event={"ID":"1f688849-f103-4589-8258-8e83a3015197","Type":"ContainerDied","Data":"33ccb910b7e0611612eb40195633c732d071dae0e900edc6f3928efcdb5bbc18"}
Sep 09 20:36:03.508833 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.508623    2563 scope.go:117] "RemoveContainer" containerID="30cf8904de98fcfe3bd5e16bc0e4a9e23b035c9ef090b36e4cd32791f9d218cb"
Sep 09 20:36:03.604073 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.604034    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:03.661262 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.661182    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:36:03.716049 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f688849-f103-4589-8258-8e83a3015197-istio-token\") pod \"1f688849-f103-4589-8258-8e83a3015197\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") "
Sep 09 20:36:03.716245 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716059    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-credential-socket\") pod \"1f688849-f103-4589-8258-8e83a3015197\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") "
Sep 09 20:36:03.716245 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716081    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-workload-certs\") pod \"1f688849-f103-4589-8258-8e83a3015197\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") "
Sep 09 20:36:03.716245 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716123    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-istio-data\") pod \"1f688849-f103-4589-8258-8e83a3015197\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") "
Sep 09 20:36:03.716245 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716156    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f688849-f103-4589-8258-8e83a3015197-istio-podinfo\") pod \"1f688849-f103-4589-8258-8e83a3015197\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") "
Sep 09 20:36:03.716245 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716179    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-istio-envoy\") pod \"1f688849-f103-4589-8258-8e83a3015197\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") "
Sep 09 20:36:03.716245 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f688849-f103-4589-8258-8e83a3015197-istiod-ca-cert\") pod \"1f688849-f103-4589-8258-8e83a3015197\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") "
Sep 09 20:36:03.716601 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716255    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-workload-socket\") pod \"1f688849-f103-4589-8258-8e83a3015197\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") "
Sep 09 20:36:03.716601 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716287    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k47ks\" (UniqueName: \"kubernetes.io/projected/1f688849-f103-4589-8258-8e83a3015197-kube-api-access-k47ks\") pod \"1f688849-f103-4589-8258-8e83a3015197\" (UID: \"1f688849-f103-4589-8258-8e83a3015197\") "
Sep 09 20:36:03.716601 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716366    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1f688849-f103-4589-8258-8e83a3015197" (UID: "1f688849-f103-4589-8258-8e83a3015197"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:03.716601 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716528    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-istio-data" (OuterVolumeSpecName: "istio-data") pod "1f688849-f103-4589-8258-8e83a3015197" (UID: "1f688849-f103-4589-8258-8e83a3015197"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:03.716819 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716604    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:36:03.716819 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716660    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1f688849-f103-4589-8258-8e83a3015197" (UID: "1f688849-f103-4589-8258-8e83a3015197"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:03.716819 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716652    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f688849-f103-4589-8258-8e83a3015197-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1f688849-f103-4589-8258-8e83a3015197" (UID: "1f688849-f103-4589-8258-8e83a3015197"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:03.716819 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.716797    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1f688849-f103-4589-8258-8e83a3015197" (UID: "1f688849-f103-4589-8258-8e83a3015197"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:03.718455 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.718426    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1f688849-f103-4589-8258-8e83a3015197" (UID: "1f688849-f103-4589-8258-8e83a3015197"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:03.718455 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.718436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f688849-f103-4589-8258-8e83a3015197-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1f688849-f103-4589-8258-8e83a3015197" (UID: "1f688849-f103-4589-8258-8e83a3015197"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:36:03.718754 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.718736    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f688849-f103-4589-8258-8e83a3015197-istio-token" (OuterVolumeSpecName: "istio-token") pod "1f688849-f103-4589-8258-8e83a3015197" (UID: "1f688849-f103-4589-8258-8e83a3015197"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:03.718814 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.718788    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f688849-f103-4589-8258-8e83a3015197-kube-api-access-k47ks" (OuterVolumeSpecName: "kube-api-access-k47ks") pod "1f688849-f103-4589-8258-8e83a3015197" (UID: "1f688849-f103-4589-8258-8e83a3015197"). InnerVolumeSpecName "kube-api-access-k47ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:03.817179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.817120    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:36:03.817179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.817173    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:36:03.817179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.817182    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f688849-f103-4589-8258-8e83a3015197-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:36:03.817179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.817190    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:36:03.817179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.817199    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f688849-f103-4589-8258-8e83a3015197-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:36:03.817541 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.817208    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f688849-f103-4589-8258-8e83a3015197-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:36:03.817541 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.817217    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k47ks\" (UniqueName: \"kubernetes.io/projected/1f688849-f103-4589-8258-8e83a3015197-kube-api-access-k47ks\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:36:03.817541 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:03.817227    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f688849-f103-4589-8258-8e83a3015197-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:36:04.514901 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:04.514869    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"
Sep 09 20:36:04.515321 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:04.514870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c" event={"ID":"1f688849-f103-4589-8258-8e83a3015197","Type":"ContainerDied","Data":"cd162f8439812a069c3d8cac6bd66f8c453c1e29f919e9d37ddcfadd2a5df708"}
Sep 09 20:36:04.515321 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:04.515011    2563 scope.go:117] "RemoveContainer" containerID="33ccb910b7e0611612eb40195633c732d071dae0e900edc6f3928efcdb5bbc18"
Sep 09 20:36:04.571929 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:04.571889    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"]
Sep 09 20:36:04.588166 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:04.588125    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4defd2-openshift-default-5df668f74b-cz77c"]
Sep 09 20:36:04.605393 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:04.605358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:04.727613 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:04.727553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:05.222074 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.222038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:05.313408 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.313372    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f688849-f103-4589-8258-8e83a3015197" path="/var/lib/kubelet/pods/1f688849-f103-4589-8258-8e83a3015197/volumes"
Sep 09 20:36:05.319118 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.319091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:05.349840 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.349805    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"]
Sep 09 20:36:05.350150 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.350133    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ddea5f1-7cb2-4df5-ae87-1f900be04a67" containerName="payload-pre-processing"
Sep 09 20:36:05.350150 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.350151    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ddea5f1-7cb2-4df5-ae87-1f900be04a67" containerName="payload-pre-processing"
Sep 09 20:36:05.350284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.350162    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy"
Sep 09 20:36:05.350284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.350168    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy"
Sep 09 20:36:05.350284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.350176    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy"
Sep 09 20:36:05.350284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.350181    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy"
Sep 09 20:36:05.350284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.350193    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy"
Sep 09 20:36:05.350284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.350198    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1314a2b3-c135-4998-99b5-76a9b81370c2" containerName="istio-proxy"
Sep 09 20:36:05.350284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.350248    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ddea5f1-7cb2-4df5-ae87-1f900be04a67" containerName="payload-pre-processing"
Sep 09 20:36:05.350284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.350256    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy"
Sep 09 20:36:05.350284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.350263    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f688849-f103-4589-8258-8e83a3015197" containerName="istio-proxy"
Sep 09 20:36:05.354795 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.354778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"
Sep 09 20:36:05.357216 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.357194    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-166b94\""
Sep 09 20:36:05.357341 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.357217    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-166b94-dockercfg-mltqj\""
Sep 09 20:36:05.365031 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.365005    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"]
Sep 09 20:36:05.431186 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.431140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbf3e43-5a97-4dd7-8301-2ac711f8d08c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h\" (UID: \"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"
Sep 09 20:36:05.431346 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.431198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ht65\" (UniqueName: \"kubernetes.io/projected/8dbf3e43-5a97-4dd7-8301-2ac711f8d08c-kube-api-access-2ht65\") pod \"payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h\" (UID: \"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"
Sep 09 20:36:05.532461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.532362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbf3e43-5a97-4dd7-8301-2ac711f8d08c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h\" (UID: \"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"
Sep 09 20:36:05.532461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.532412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ht65\" (UniqueName: \"kubernetes.io/projected/8dbf3e43-5a97-4dd7-8301-2ac711f8d08c-kube-api-access-2ht65\") pod \"payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h\" (UID: \"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"
Sep 09 20:36:05.533249 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.533221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbf3e43-5a97-4dd7-8301-2ac711f8d08c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h\" (UID: \"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"
Sep 09 20:36:05.541340 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.541315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ht65\" (UniqueName: \"kubernetes.io/projected/8dbf3e43-5a97-4dd7-8301-2ac711f8d08c-kube-api-access-2ht65\") pod \"payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h\" (UID: \"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"
Sep 09 20:36:05.666902 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.666863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"
Sep 09 20:36:05.668552 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.668507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:05.939435 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:05.939307    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"]
Sep 09 20:36:05.941927 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:36:05.941897    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbf3e43_5a97_4dd7_8301_2ac711f8d08c.slice/crio-f53177113d2a59c8a721fbd4633a75edf973eec4acd1ff7a9a57681fbfec8b67 WatchSource:0}: Error finding container f53177113d2a59c8a721fbd4633a75edf973eec4acd1ff7a9a57681fbfec8b67: Status 404 returned error can't find the container with id f53177113d2a59c8a721fbd4633a75edf973eec4acd1ff7a9a57681fbfec8b67
Sep 09 20:36:06.526952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:06.526910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:06.527965 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:06.527943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h" event={"ID":"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c","Type":"ContainerStarted","Data":"0f5803536a397339ce4ef55a9966efc31c7904611ad6e893a16e421b4c8ce014"}
Sep 09 20:36:06.528092 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:06.527987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h" event={"ID":"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c","Type":"ContainerStarted","Data":"f53177113d2a59c8a721fbd4633a75edf973eec4acd1ff7a9a57681fbfec8b67"}
Sep 09 20:36:06.528092 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:06.528022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"
Sep 09 20:36:06.546438 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:06.546382    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h" podStartSLOduration=1.5463621509999999 podStartE2EDuration="1.546362151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:06.543931732 +0000 UTC m=+1173.783034059" watchObservedRunningTime="2026-09-09 20:36:06.546362151 +0000 UTC m=+1173.785464453"
Sep 09 20:36:06.593050 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:06.593013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:07.298689 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:07.298647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:07.408465 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:07.408435    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7577-6389d7"]
Sep 09 20:36:07.411799 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:07.411783    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-6389d7"
Sep 09 20:36:07.413928 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:07.413908    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5fctx\""
Sep 09 20:36:07.417276 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:07.417254    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7577-6389d7"]
Sep 09 20:36:07.551365 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:07.551275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r752\" (UniqueName: \"kubernetes.io/projected/dd030c62-952e-460d-bbe0-01ee52c2845c-kube-api-access-7r752\") pod \"test-curl-7577-6389d7\" (UID: \"dd030c62-952e-460d-bbe0-01ee52c2845c\") " pod="opendatahub/test-curl-7577-6389d7"
Sep 09 20:36:07.652470 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:07.652426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r752\" (UniqueName: \"kubernetes.io/projected/dd030c62-952e-460d-bbe0-01ee52c2845c-kube-api-access-7r752\") pod \"test-curl-7577-6389d7\" (UID: \"dd030c62-952e-460d-bbe0-01ee52c2845c\") " pod="opendatahub/test-curl-7577-6389d7"
Sep 09 20:36:07.660187 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:07.660154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r752\" (UniqueName: \"kubernetes.io/projected/dd030c62-952e-460d-bbe0-01ee52c2845c-kube-api-access-7r752\") pod \"test-curl-7577-6389d7\" (UID: \"dd030c62-952e-460d-bbe0-01ee52c2845c\") " pod="opendatahub/test-curl-7577-6389d7"
Sep 09 20:36:07.723020 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:07.722980    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-6389d7"
Sep 09 20:36:07.992593 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:07.992540    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7577-6389d7"]
Sep 09 20:36:07.996857 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:36:07.996794    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd030c62_952e_460d_bbe0_01ee52c2845c.slice/crio-7db193ef6be25bba2dc086681116021aa8bd3eeb2568df026051add96c9fc539 WatchSource:0}: Error finding container 7db193ef6be25bba2dc086681116021aa8bd3eeb2568df026051add96c9fc539: Status 404 returned error can't find the container with id 7db193ef6be25bba2dc086681116021aa8bd3eeb2568df026051add96c9fc539
Sep 09 20:36:07.999618 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:07.999477    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:36:08.108493 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:08.108454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:08.539755 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:08.539712    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7577-6389d7" event={"ID":"dd030c62-952e-460d-bbe0-01ee52c2845c","Type":"ContainerStarted","Data":"bb277506e4b6551bda288b08dd568c8842e9689dd8efdda800ba8280189b0a28"}
Sep 09 20:36:08.539755 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:08.539762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7577-6389d7" event={"ID":"dd030c62-952e-460d-bbe0-01ee52c2845c","Type":"ContainerStarted","Data":"7db193ef6be25bba2dc086681116021aa8bd3eeb2568df026051add96c9fc539"}
Sep 09 20:36:08.556283 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:08.556233    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7577-6389d7" podStartSLOduration=1.556218094 podStartE2EDuration="1.556218094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:08.555950468 +0000 UTC m=+1175.795052768" watchObservedRunningTime="2026-09-09 20:36:08.556218094 +0000 UTC m=+1175.795320393"
Sep 09 20:36:09.203194 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.203158    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7577-6389d7"]
Sep 09 20:36:09.207452 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.207422    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7577-6389d7"]
Sep 09 20:36:09.347122 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.347040    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7577-a730b0"]
Sep 09 20:36:09.347384 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.347371    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd030c62-952e-460d-bbe0-01ee52c2845c" containerName="test-curl-7577-6389d7"
Sep 09 20:36:09.347433 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.347385    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd030c62-952e-460d-bbe0-01ee52c2845c" containerName="test-curl-7577-6389d7"
Sep 09 20:36:09.347469 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.347441    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd030c62-952e-460d-bbe0-01ee52c2845c" containerName="test-curl-7577-6389d7"
Sep 09 20:36:09.350546 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.350527    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-a730b0"
Sep 09 20:36:09.358900 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.358190    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7577-a730b0"]
Sep 09 20:36:09.469321 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.469272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb48p\" (UniqueName: \"kubernetes.io/projected/bd62c924-8ef3-4a05-a08e-6725f1771c58-kube-api-access-bb48p\") pod \"test-curl-7577-a730b0\" (UID: \"bd62c924-8ef3-4a05-a08e-6725f1771c58\") " pod="opendatahub/test-curl-7577-a730b0"
Sep 09 20:36:09.544632 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:36:09.544600    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7577-6389d7" container="test-curl-7577-6389d7"
Sep 09 20:36:09.570723 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.570686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb48p\" (UniqueName: \"kubernetes.io/projected/bd62c924-8ef3-4a05-a08e-6725f1771c58-kube-api-access-bb48p\") pod \"test-curl-7577-a730b0\" (UID: \"bd62c924-8ef3-4a05-a08e-6725f1771c58\") " pod="opendatahub/test-curl-7577-a730b0"
Sep 09 20:36:09.579375 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.579347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb48p\" (UniqueName: \"kubernetes.io/projected/bd62c924-8ef3-4a05-a08e-6725f1771c58-kube-api-access-bb48p\") pod \"test-curl-7577-a730b0\" (UID: \"bd62c924-8ef3-4a05-a08e-6725f1771c58\") " pod="opendatahub/test-curl-7577-a730b0"
Sep 09 20:36:09.667654 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.667539    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-a730b0"
Sep 09 20:36:09.932895 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.932844    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7577-a730b0"]
Sep 09 20:36:09.935474 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:36:09.935441    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd62c924_8ef3_4a05_a08e_6725f1771c58.slice/crio-50de8d68c03b940e467e7693a9fce2365851e47b628bca7fdada1da8f7207305 WatchSource:0}: Error finding container 50de8d68c03b940e467e7693a9fce2365851e47b628bca7fdada1da8f7207305: Status 404 returned error can't find the container with id 50de8d68c03b940e467e7693a9fce2365851e47b628bca7fdada1da8f7207305
Sep 09 20:36:09.960909 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:09.960862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:10.551221 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:10.551177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7577-a730b0" event={"ID":"bd62c924-8ef3-4a05-a08e-6725f1771c58","Type":"ContainerStarted","Data":"1a1fac54d3b25091d91f116ca244edf6fd7d071c478f70cbcbab2fd11f9cbb9d"}
Sep 09 20:36:10.551221 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:10.551227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7577-a730b0" event={"ID":"bd62c924-8ef3-4a05-a08e-6725f1771c58","Type":"ContainerStarted","Data":"50de8d68c03b940e467e7693a9fce2365851e47b628bca7fdada1da8f7207305"}
Sep 09 20:36:10.551547 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:10.551355    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7577-6389d7" podUID="dd030c62-952e-460d-bbe0-01ee52c2845c" containerName="test-curl-7577-6389d7" containerID="cri-o://bb277506e4b6551bda288b08dd568c8842e9689dd8efdda800ba8280189b0a28" gracePeriod=2
Sep 09 20:36:10.564967 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:10.564909    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7577-a730b0" podStartSLOduration=1.564892286 podStartE2EDuration="1.564892286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:10.563840001 +0000 UTC m=+1177.802942302" watchObservedRunningTime="2026-09-09 20:36:10.564892286 +0000 UTC m=+1177.803994608"
Sep 09 20:36:11.228867 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:11.228776    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7577-a730b0"]
Sep 09 20:36:11.232334 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:11.232305    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7577-a730b0"]
Sep 09 20:36:11.555801 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:36:11.555717    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7577-a730b0" container="test-curl-7577-a730b0"
Sep 09 20:36:12.560828 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:12.560778    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7577-a730b0" podUID="bd62c924-8ef3-4a05-a08e-6725f1771c58" containerName="test-curl-7577-a730b0" containerID="cri-o://1a1fac54d3b25091d91f116ca244edf6fd7d071c478f70cbcbab2fd11f9cbb9d" gracePeriod=2
Sep 09 20:36:12.724163 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:12.724134    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-6389d7"
Sep 09 20:36:12.800480 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:12.800432    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r752\" (UniqueName: \"kubernetes.io/projected/dd030c62-952e-460d-bbe0-01ee52c2845c-kube-api-access-7r752\") pod \"dd030c62-952e-460d-bbe0-01ee52c2845c\" (UID: \"dd030c62-952e-460d-bbe0-01ee52c2845c\") "
Sep 09 20:36:12.802545 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:12.802517    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd030c62-952e-460d-bbe0-01ee52c2845c-kube-api-access-7r752" (OuterVolumeSpecName: "kube-api-access-7r752") pod "dd030c62-952e-460d-bbe0-01ee52c2845c" (UID: "dd030c62-952e-460d-bbe0-01ee52c2845c"). InnerVolumeSpecName "kube-api-access-7r752". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:12.901557 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:12.901518    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r752\" (UniqueName: \"kubernetes.io/projected/dd030c62-952e-460d-bbe0-01ee52c2845c-kube-api-access-7r752\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:36:13.312960 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:13.312924    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd030c62-952e-460d-bbe0-01ee52c2845c" path="/var/lib/kubelet/pods/dd030c62-952e-460d-bbe0-01ee52c2845c/volumes"
Sep 09 20:36:13.565971 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:13.565881    2563 generic.go:356] "Generic (PLEG): container finished" podID="dd030c62-952e-460d-bbe0-01ee52c2845c" containerID="bb277506e4b6551bda288b08dd568c8842e9689dd8efdda800ba8280189b0a28" exitCode=137
Sep 09 20:36:13.565971 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:13.565925    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-6389d7"
Sep 09 20:36:13.565971 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:13.565967    2563 scope.go:117] "RemoveContainer" containerID="bb277506e4b6551bda288b08dd568c8842e9689dd8efdda800ba8280189b0a28"
Sep 09 20:36:13.574227 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:13.574206    2563 scope.go:117] "RemoveContainer" containerID="bb277506e4b6551bda288b08dd568c8842e9689dd8efdda800ba8280189b0a28"
Sep 09 20:36:13.574512 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:36:13.574493    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb277506e4b6551bda288b08dd568c8842e9689dd8efdda800ba8280189b0a28\": container with ID starting with bb277506e4b6551bda288b08dd568c8842e9689dd8efdda800ba8280189b0a28 not found: ID does not exist" containerID="bb277506e4b6551bda288b08dd568c8842e9689dd8efdda800ba8280189b0a28"
Sep 09 20:36:13.574591 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:13.574521    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb277506e4b6551bda288b08dd568c8842e9689dd8efdda800ba8280189b0a28"} err="failed to get container status \"bb277506e4b6551bda288b08dd568c8842e9689dd8efdda800ba8280189b0a28\": rpc error: code = NotFound desc = could not find container \"bb277506e4b6551bda288b08dd568c8842e9689dd8efdda800ba8280189b0a28\": container with ID starting with bb277506e4b6551bda288b08dd568c8842e9689dd8efdda800ba8280189b0a28 not found: ID does not exist"
Sep 09 20:36:14.704819 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:14.704789    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-a730b0"
Sep 09 20:36:14.817848 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:14.817749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bb48p\" (UniqueName: \"kubernetes.io/projected/bd62c924-8ef3-4a05-a08e-6725f1771c58-kube-api-access-bb48p\") pod \"bd62c924-8ef3-4a05-a08e-6725f1771c58\" (UID: \"bd62c924-8ef3-4a05-a08e-6725f1771c58\") "
Sep 09 20:36:14.819855 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:14.819825    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd62c924-8ef3-4a05-a08e-6725f1771c58-kube-api-access-bb48p" (OuterVolumeSpecName: "kube-api-access-bb48p") pod "bd62c924-8ef3-4a05-a08e-6725f1771c58" (UID: "bd62c924-8ef3-4a05-a08e-6725f1771c58"). InnerVolumeSpecName "kube-api-access-bb48p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:14.919007 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:14.918969    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb48p\" (UniqueName: \"kubernetes.io/projected/bd62c924-8ef3-4a05-a08e-6725f1771c58-kube-api-access-bb48p\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:36:15.313586 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:15.313546    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd62c924-8ef3-4a05-a08e-6725f1771c58" path="/var/lib/kubelet/pods/bd62c924-8ef3-4a05-a08e-6725f1771c58/volumes"
Sep 09 20:36:15.577976 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:15.577875    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd62c924-8ef3-4a05-a08e-6725f1771c58" containerID="1a1fac54d3b25091d91f116ca244edf6fd7d071c478f70cbcbab2fd11f9cbb9d" exitCode=137
Sep 09 20:36:15.577976 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:15.577915    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-a730b0"
Sep 09 20:36:15.578193 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:15.577982    2563 scope.go:117] "RemoveContainer" containerID="1a1fac54d3b25091d91f116ca244edf6fd7d071c478f70cbcbab2fd11f9cbb9d"
Sep 09 20:36:15.586356 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:15.586192    2563 scope.go:117] "RemoveContainer" containerID="1a1fac54d3b25091d91f116ca244edf6fd7d071c478f70cbcbab2fd11f9cbb9d"
Sep 09 20:36:15.586591 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:36:15.586551    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1fac54d3b25091d91f116ca244edf6fd7d071c478f70cbcbab2fd11f9cbb9d\": container with ID starting with 1a1fac54d3b25091d91f116ca244edf6fd7d071c478f70cbcbab2fd11f9cbb9d not found: ID does not exist" containerID="1a1fac54d3b25091d91f116ca244edf6fd7d071c478f70cbcbab2fd11f9cbb9d"
Sep 09 20:36:15.586647 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:15.586596    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1fac54d3b25091d91f116ca244edf6fd7d071c478f70cbcbab2fd11f9cbb9d"} err="failed to get container status \"1a1fac54d3b25091d91f116ca244edf6fd7d071c478f70cbcbab2fd11f9cbb9d\": rpc error: code = NotFound desc = could not find container \"1a1fac54d3b25091d91f116ca244edf6fd7d071c478f70cbcbab2fd11f9cbb9d\": container with ID starting with 1a1fac54d3b25091d91f116ca244edf6fd7d071c478f70cbcbab2fd11f9cbb9d not found: ID does not exist"
Sep 09 20:36:15.708299 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:15.708247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:16.054354 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.054317    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"]
Sep 09 20:36:16.054733 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.054716    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd62c924-8ef3-4a05-a08e-6725f1771c58" containerName="test-curl-7577-a730b0"
Sep 09 20:36:16.054733 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.054733    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd62c924-8ef3-4a05-a08e-6725f1771c58" containerName="test-curl-7577-a730b0"
Sep 09 20:36:16.054881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.054799    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd62c924-8ef3-4a05-a08e-6725f1771c58" containerName="test-curl-7577-a730b0"
Sep 09 20:36:16.059166 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.059142    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.063248 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.063223    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-57f5cbb5-openshift-default-dockercfg-nc8ns\""
Sep 09 20:36:16.082548 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.082489    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"]
Sep 09 20:36:16.112264 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.112227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:16.130579 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.130531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-credential-socket\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.130758 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.130618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-workload-socket\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.130758 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.130650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-envoy\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.130758 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.130675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwqmv\" (UniqueName: \"kubernetes.io/projected/460d3a78-6e5f-4411-ab2f-190ac3613348-kube-api-access-jwqmv\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.130903 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.130776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-token\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.130903 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.130812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460d3a78-6e5f-4411-ab2f-190ac3613348-istiod-ca-cert\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.130987 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.130908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-workload-certs\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.130987 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.130952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-podinfo\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.130987 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.130972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-data\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.231658 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.231607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-podinfo\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.231658 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.231648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-data\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.231912 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.231766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-credential-socket\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.231912 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.231832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-workload-socket\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.231912 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.231852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-envoy\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.231912 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.231873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwqmv\" (UniqueName: \"kubernetes.io/projected/460d3a78-6e5f-4411-ab2f-190ac3613348-kube-api-access-jwqmv\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.232126 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.231965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-token\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.232126 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.232016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460d3a78-6e5f-4411-ab2f-190ac3613348-istiod-ca-cert\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.232126 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.232025    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-data\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.232126 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.232105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-workload-certs\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.232329 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.232119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-credential-socket\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.232329 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.232178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-workload-socket\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.232427 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.232346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-workload-certs\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.232697 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.232676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460d3a78-6e5f-4411-ab2f-190ac3613348-istiod-ca-cert\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.234270 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.234251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-podinfo\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.234352 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.234274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-envoy\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.239745 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.239718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-token\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.239953 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.239933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwqmv\" (UniqueName: \"kubernetes.io/projected/460d3a78-6e5f-4411-ab2f-190ac3613348-kube-api-access-jwqmv\") pod \"e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") " pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.370402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.370296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:16.653597 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.653544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"]
Sep 09 20:36:16.655957 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:36:16.655924    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460d3a78_6e5f_4411_ab2f_190ac3613348.slice/crio-462dc6765912c05e504c4d93afdce4f2c7704be3137492b6dff7892aa6b70830 WatchSource:0}: Error finding container 462dc6765912c05e504c4d93afdce4f2c7704be3137492b6dff7892aa6b70830: Status 404 returned error can't find the container with id 462dc6765912c05e504c4d93afdce4f2c7704be3137492b6dff7892aa6b70830
Sep 09 20:36:16.661734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.661700    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:36:16.661832 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.661782    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:36:16.661911 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.661831    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:36:16.675813 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:16.675766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:17.535014 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:17.534983    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"
Sep 09 20:36:17.591749 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:17.591715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" event={"ID":"460d3a78-6e5f-4411-ab2f-190ac3613348","Type":"ContainerStarted","Data":"4992638228d40527dab9d6d7ae3c18471e6a0b1f9ec56ff289c3285307aadfa4"}
Sep 09 20:36:17.591913 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:17.591770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" event={"ID":"460d3a78-6e5f-4411-ab2f-190ac3613348","Type":"ContainerStarted","Data":"462dc6765912c05e504c4d93afdce4f2c7704be3137492b6dff7892aa6b70830"}
Sep 09 20:36:17.611032 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:17.610978    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" podStartSLOduration=1.6109614749999999 podStartE2EDuration="1.610961475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:17.609334981 +0000 UTC m=+1184.848437281" watchObservedRunningTime="2026-09-09 20:36:17.610961475 +0000 UTC m=+1184.850063775"
Sep 09 20:36:18.371063 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:18.371015    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:18.376085 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:18.376060    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:18.597016 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:18.596979    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:18.598069 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:18.598048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:19.161838 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:19.161791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:19.807328 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:19.807274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:19.903789 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:19.903753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:20.170840 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:20.170744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:20.913735 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:20.913692    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:21.700162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:21.700129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:22.469235 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.469200    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"]
Sep 09 20:36:22.472920 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.472896    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:22.475460 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.475435    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-57f5cbb5-serving-cert\""
Sep 09 20:36:22.485781 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.485750    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"]
Sep 09 20:36:22.504671 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.504642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:22.566201 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.566161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:22.587804 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.587765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-maas-api-tls\") pod \"maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:22.588164 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.588117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-metrics-certs\") pod \"maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:22.588423 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.588405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bbh9\" (UniqueName: \"kubernetes.io/projected/380051bc-7985-46d5-9582-51ac0f89c556-kube-api-access-6bbh9\") pod \"maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:22.588642 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.588622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380051bc-7985-46d5-9582-51ac0f89c556-openshift-service-ca\") pod \"maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:22.689762 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.689725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380051bc-7985-46d5-9582-51ac0f89c556-openshift-service-ca\") pod \"maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:22.689968 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.689801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-maas-api-tls\") pod \"maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:22.689968 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.689830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-metrics-certs\") pod \"maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:22.689968 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.689874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bbh9\" (UniqueName: \"kubernetes.io/projected/380051bc-7985-46d5-9582-51ac0f89c556-kube-api-access-6bbh9\") pod \"maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:22.690438 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:36:22.690416    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-serving-cert: secret "maas-api-e2e-mt-57f5cbb5-serving-cert" not found
Sep 09 20:36:22.690536 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:36:22.690479    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-maas-api-tls podName:380051bc-7985-46d5-9582-51ac0f89c556 nodeName:}" failed. No retries permitted until 2026-09-09 20:36:23.190465516 +0000 UTC m=+1190.429567794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-maas-api-tls") pod "maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz" (UID: "380051bc-7985-46d5-9582-51ac0f89c556") : secret "maas-api-e2e-mt-57f5cbb5-serving-cert" not found
Sep 09 20:36:22.690612 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.690553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380051bc-7985-46d5-9582-51ac0f89c556-openshift-service-ca\") pod \"maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:22.692434 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.692408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-metrics-certs\") pod \"maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:22.698240 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.698213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bbh9\" (UniqueName: \"kubernetes.io/projected/380051bc-7985-46d5-9582-51ac0f89c556-kube-api-access-6bbh9\") pod \"maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:22.960812 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:22.960780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:23.105940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:23.105904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:23.194677 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:23.194632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-maas-api-tls\") pod \"maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:23.197113 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:23.197087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-maas-api-tls\") pod \"maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:23.199625 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:23.199600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:23.387697 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:23.387590    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:23.512657 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:23.512621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:23.658976 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:23.658907    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"]
Sep 09 20:36:23.661137 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:36:23.661110    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380051bc_7985_46d5_9582_51ac0f89c556.slice/crio-0a6fb80e52873c59c4a93dccb8ac7f6ea109a55c76445955e58ca773ed4f9dc0 WatchSource:0}: Error finding container 0a6fb80e52873c59c4a93dccb8ac7f6ea109a55c76445955e58ca773ed4f9dc0: Status 404 returned error can't find the container with id 0a6fb80e52873c59c4a93dccb8ac7f6ea109a55c76445955e58ca773ed4f9dc0
Sep 09 20:36:24.377738 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:24.377700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:24.630387 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:24.630305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz" event={"ID":"380051bc-7985-46d5-9582-51ac0f89c556","Type":"ContainerStarted","Data":"56fc8b4da818840edcf4ab084dc924c91611df27e90d3f94653a27f38b6c63c3"}
Sep 09 20:36:24.630387 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:24.630356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz" event={"ID":"380051bc-7985-46d5-9582-51ac0f89c556","Type":"ContainerStarted","Data":"0a6fb80e52873c59c4a93dccb8ac7f6ea109a55c76445955e58ca773ed4f9dc0"}
Sep 09 20:36:24.630823 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:24.630398    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:24.648729 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:24.648677    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz" podStartSLOduration=2.089401624 podStartE2EDuration="2.648661566s" totalImagesPullingTime="559.259942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:24.647025559 +0000 UTC m=+1191.886127860" watchObservedRunningTime="2026-09-09 20:36:24.648661566 +0000 UTC m=+1191.887763922"
Sep 09 20:36:25.072794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:25.072753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:25.722968 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:25.722934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:26.304255 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:26.304220    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:36:26.459150 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:26.459107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:27.267048 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:27.267012    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:27.798780 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:27.798741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:28.151425 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.151330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:28.466381 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.466353    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"]
Sep 09 20:36:28.483885 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.483850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.484984 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.484954    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"]
Sep 09 20:36:28.486342 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.486316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:28.487085 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.487064    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-4101f2-openshift-default-dockercfg-w22z7\""
Sep 09 20:36:28.656363 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.656318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-workload-socket\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.656606 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.656373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-token\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.656606 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.656422    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-data\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.656606 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.656465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-workload-certs\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.656606 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.656549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xqcv\" (UniqueName: \"kubernetes.io/projected/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-kube-api-access-6xqcv\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.656606 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.656593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-credential-socket\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.656890 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.656618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-envoy\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.656890 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.656655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-podinfo\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.656890 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.656684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istiod-ca-cert\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.697772 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.697739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:28.758109 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.758013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-workload-certs\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.758385 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.758366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xqcv\" (UniqueName: \"kubernetes.io/projected/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-kube-api-access-6xqcv\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.758511 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.758495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-credential-socket\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.758633 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.758616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-envoy\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.758751 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.758739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-podinfo\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.758845 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.758833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istiod-ca-cert\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.758959 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.758944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-workload-socket\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.759049 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.759037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-token\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.759165 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.759153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-data\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.759296 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.758503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-workload-certs\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.760116 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.759055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-credential-socket\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.760296 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.760230    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-workload-socket\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.760296 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.760240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istiod-ca-cert\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.760479 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.760362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-data\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.763051 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.763020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-envoy\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.765509 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.765461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-podinfo\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.767804 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.767756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xqcv\" (UniqueName: \"kubernetes.io/projected/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-kube-api-access-6xqcv\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.768360 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.768320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-token\") pod \"e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") " pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:28.797863 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:28.797818    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:29.099816 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:29.099792    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"]
Sep 09 20:36:29.101928 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:36:29.101902    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb1fbbc_7daf_46a3_8595_02f4f6b2a9fe.slice/crio-259c70d5c55428820d232a44899171ab6959254d9ecfbf391d7295d051117b3d WatchSource:0}: Error finding container 259c70d5c55428820d232a44899171ab6959254d9ecfbf391d7295d051117b3d: Status 404 returned error can't find the container with id 259c70d5c55428820d232a44899171ab6959254d9ecfbf391d7295d051117b3d
Sep 09 20:36:29.104488 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:29.104447    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:36:29.104556 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:29.104540    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:36:29.104629 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:29.104613    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:36:29.662523 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:29.662476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" event={"ID":"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe","Type":"ContainerStarted","Data":"ea030676ed85f612b4350c1796f1eda6b6b1a59bac098ef6852b60c1e1744e0c"}
Sep 09 20:36:29.663013 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:29.662554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" event={"ID":"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe","Type":"ContainerStarted","Data":"259c70d5c55428820d232a44899171ab6959254d9ecfbf391d7295d051117b3d"}
Sep 09 20:36:29.683037 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:29.682977    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" podStartSLOduration=1.6829594 podStartE2EDuration="1.6829594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:29.681939023 +0000 UTC m=+1196.921041323" watchObservedRunningTime="2026-09-09 20:36:29.6829594 +0000 UTC m=+1196.922061699"
Sep 09 20:36:29.798592 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:29.798546    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:30.640609 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:30.640557    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:36:30.803507 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:30.803477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:30.874778 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:30.874743    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:31.575516 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:31.575473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:31.673771 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:31.673723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:31.676326 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:31.676306    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:31.677495 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:31.677474    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:32.460535 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:32.460488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:33.257960 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:33.257920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:33.324555 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:33.324528    2563 scope.go:117] "RemoveContainer" containerID="c31b9eea1c51a192c069f80d0665561733afe4355f0a90854f781408f1874def"
Sep 09 20:36:34.083337 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:34.083290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:34.501055 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:34.500995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:34.752122 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:34.752015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:35.418790 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:35.418755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:35.520172 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:35.520120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:35.621132 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:35.621048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:35.888095 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:35.888055    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:35.974412 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:35.974365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:36.478619 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:36.478566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:37.173089 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:37.172999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:37.708514 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:37.708482    2563 generic.go:356] "Generic (PLEG): container finished" podID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerID="4992638228d40527dab9d6d7ae3c18471e6a0b1f9ec56ff289c3285307aadfa4" exitCode=0
Sep 09 20:36:37.708931 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:37.708564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" event={"ID":"460d3a78-6e5f-4411-ab2f-190ac3613348","Type":"ContainerDied","Data":"4992638228d40527dab9d6d7ae3c18471e6a0b1f9ec56ff289c3285307aadfa4"}
Sep 09 20:36:37.708984 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:37.708973    2563 scope.go:117] "RemoveContainer" containerID="4992638228d40527dab9d6d7ae3c18471e6a0b1f9ec56ff289c3285307aadfa4"
Sep 09 20:36:37.710080 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:37.710052    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:36:37.710135 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:37.710122    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:36:37.710170 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:37.710152    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:36:38.716383 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:38.716350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" event={"ID":"460d3a78-6e5f-4411-ab2f-190ac3613348","Type":"ContainerStarted","Data":"a35389844cb07c75f86024c01b844c231d9ed991951e267f9567f80e60ffbc51"}
Sep 09 20:36:39.371168 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:39.371116    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:39.371501 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:39.371472    2563 patch_prober.go:28] interesting pod/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 09 20:36:39.371605 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:39.371531    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 09 20:36:40.034232 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:40.034196    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:40.370914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:40.370816    2563 patch_prober.go:28] interesting pod/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 09 20:36:40.370914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:40.370899    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 09 20:36:41.370982 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:41.370941    2563 patch_prober.go:28] interesting pod/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 09 20:36:41.371407 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:41.371013    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 09 20:36:41.840598 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:41.840533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:41.940848 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:41.940813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:42.210802 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:42.210764    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:36:42.370776 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:42.370730    2563 patch_prober.go:28] interesting pod/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 09 20:36:42.370981 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:42.370803    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 09 20:36:43.371359 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:43.371319    2563 patch_prober.go:28] interesting pod/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 09 20:36:43.371814 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:43.371408    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 09 20:36:43.584131 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:43.584081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:44.371732 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:44.371692    2563 patch_prober.go:28] interesting pod/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 09 20:36:44.372335 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:44.372299    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 09 20:36:45.376270 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:45.376238    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:45.376678 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:45.376609    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:45.377312 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:45.377294    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:36:53.805001 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:53.804962    2563 generic.go:356] "Generic (PLEG): container finished" podID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerID="ea030676ed85f612b4350c1796f1eda6b6b1a59bac098ef6852b60c1e1744e0c" exitCode=0
Sep 09 20:36:53.805491 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:53.805034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" event={"ID":"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe","Type":"ContainerDied","Data":"ea030676ed85f612b4350c1796f1eda6b6b1a59bac098ef6852b60c1e1744e0c"}
Sep 09 20:36:53.805491 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:53.805427    2563 scope.go:117] "RemoveContainer" containerID="ea030676ed85f612b4350c1796f1eda6b6b1a59bac098ef6852b60c1e1744e0c"
Sep 09 20:36:53.806504 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:53.806474    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:36:53.806622 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:53.806560    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:36:53.806697 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:53.806625    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:36:54.811520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:54.811478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" event={"ID":"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe","Type":"ContainerStarted","Data":"b1d93ca98ac60866727a2eb68794c9d1e96597186dfbf8edec0ef2b9537643ab"}
Sep 09 20:36:55.798254 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:55.798196    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:36:55.798469 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:55.798426    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 09 20:36:55.798535 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:55.798472    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 09 20:36:56.316354 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:56.316313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:56.799187 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:56.799149    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 09 20:36:56.799385 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:56.799204    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 09 20:36:57.798783 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:57.798747    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 09 20:36:57.799178 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:57.798807    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 09 20:36:57.816162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:57.816130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:58.381847 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:58.381795    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:58.798890 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:58.798852    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 09 20:36:58.799363 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:58.799130    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 09 20:36:59.186509 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:59.186424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:59.798539 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:59.798504    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 09 20:36:59.798729 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:59.798585    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 09 20:36:59.871147 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:36:59.871110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:00.690598 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:00.690553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:00.802597 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:00.802552    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:37:00.802978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:00.802952    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:37:00.803662 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:00.803643    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:37:01.162468 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:01.162431    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:37:01.592006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:01.591966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:01.717789 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:01.717759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:09.167116 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:09.167085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:09.264989 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:09.264957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:09.862114 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:09.862076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:10.630481 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:10.630444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:11.307782 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:11.307744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:12.208547 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:12.208514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:12.870127 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:12.870086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:13.505020 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:13.504981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:15.499342 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:15.499299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:15.598266 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:15.598228    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:37:15.875502 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:15.875404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:19.072317 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:19.072275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:24.702609 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.702555    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"]
Sep 09 20:37:24.703399 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.703351    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz" podUID="380051bc-7985-46d5-9582-51ac0f89c556" containerName="maas-api" containerID="cri-o://56fc8b4da818840edcf4ab084dc924c91611df27e90d3f94653a27f38b6c63c3" gracePeriod=30
Sep 09 20:37:24.733028 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.732997    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:24.804128 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.804085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:24.873280 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.873247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:24.933519 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.933474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:24.954700 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.954680    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:37:24.956510 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.956486    2563 generic.go:356] "Generic (PLEG): container finished" podID="380051bc-7985-46d5-9582-51ac0f89c556" containerID="56fc8b4da818840edcf4ab084dc924c91611df27e90d3f94653a27f38b6c63c3" exitCode=0
Sep 09 20:37:24.956642 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.956528    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"
Sep 09 20:37:24.956642 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.956532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz" event={"ID":"380051bc-7985-46d5-9582-51ac0f89c556","Type":"ContainerDied","Data":"56fc8b4da818840edcf4ab084dc924c91611df27e90d3f94653a27f38b6c63c3"}
Sep 09 20:37:24.956770 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.956643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz" event={"ID":"380051bc-7985-46d5-9582-51ac0f89c556","Type":"ContainerDied","Data":"0a6fb80e52873c59c4a93dccb8ac7f6ea109a55c76445955e58ca773ed4f9dc0"}
Sep 09 20:37:24.956770 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.956683    2563 scope.go:117] "RemoveContainer" containerID="56fc8b4da818840edcf4ab084dc924c91611df27e90d3f94653a27f38b6c63c3"
Sep 09 20:37:24.965411 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.965393    2563 scope.go:117] "RemoveContainer" containerID="56fc8b4da818840edcf4ab084dc924c91611df27e90d3f94653a27f38b6c63c3"
Sep 09 20:37:24.965719 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:37:24.965696    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fc8b4da818840edcf4ab084dc924c91611df27e90d3f94653a27f38b6c63c3\": container with ID starting with 56fc8b4da818840edcf4ab084dc924c91611df27e90d3f94653a27f38b6c63c3 not found: ID does not exist" containerID="56fc8b4da818840edcf4ab084dc924c91611df27e90d3f94653a27f38b6c63c3"
Sep 09 20:37:24.965802 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.965730    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fc8b4da818840edcf4ab084dc924c91611df27e90d3f94653a27f38b6c63c3"} err="failed to get container status \"56fc8b4da818840edcf4ab084dc924c91611df27e90d3f94653a27f38b6c63c3\": rpc error: code = NotFound desc = could not find container \"56fc8b4da818840edcf4ab084dc924c91611df27e90d3f94653a27f38b6c63c3\": container with ID starting with 56fc8b4da818840edcf4ab084dc924c91611df27e90d3f94653a27f38b6c63c3 not found: ID does not exist"
Sep 09 20:37:24.990345 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.990315    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-maas-api-tls\") pod \"380051bc-7985-46d5-9582-51ac0f89c556\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") "
Sep 09 20:37:24.990345 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.990353    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380051bc-7985-46d5-9582-51ac0f89c556-openshift-service-ca\") pod \"380051bc-7985-46d5-9582-51ac0f89c556\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") "
Sep 09 20:37:24.990649 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.990384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-metrics-certs\") pod \"380051bc-7985-46d5-9582-51ac0f89c556\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") "
Sep 09 20:37:24.990649 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.990489    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bbh9\" (UniqueName: \"kubernetes.io/projected/380051bc-7985-46d5-9582-51ac0f89c556-kube-api-access-6bbh9\") pod \"380051bc-7985-46d5-9582-51ac0f89c556\" (UID: \"380051bc-7985-46d5-9582-51ac0f89c556\") "
Sep 09 20:37:24.990791 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.990746    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380051bc-7985-46d5-9582-51ac0f89c556-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "380051bc-7985-46d5-9582-51ac0f89c556" (UID: "380051bc-7985-46d5-9582-51ac0f89c556"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:24.992877 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.992843    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380051bc-7985-46d5-9582-51ac0f89c556-kube-api-access-6bbh9" (OuterVolumeSpecName: "kube-api-access-6bbh9") pod "380051bc-7985-46d5-9582-51ac0f89c556" (UID: "380051bc-7985-46d5-9582-51ac0f89c556"). InnerVolumeSpecName "kube-api-access-6bbh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:37:24.992991 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.992909    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "380051bc-7985-46d5-9582-51ac0f89c556" (UID: "380051bc-7985-46d5-9582-51ac0f89c556"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:24.992991 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.992922    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "380051bc-7985-46d5-9582-51ac0f89c556" (UID: "380051bc-7985-46d5-9582-51ac0f89c556"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:24.997696 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:24.997533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:25.091903 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:25.091868    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:37:25.091903 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:25.091901    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380051bc-7985-46d5-9582-51ac0f89c556-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:37:25.091903 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:25.091916    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380051bc-7985-46d5-9582-51ac0f89c556-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:37:25.092189 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:25.091931    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bbh9\" (UniqueName: \"kubernetes.io/projected/380051bc-7985-46d5-9582-51ac0f89c556-kube-api-access-6bbh9\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:37:25.280446 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:25.280410    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"]
Sep 09 20:37:25.283298 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:25.283265    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57f5cbb5-77b5459b4d-llnfz"]
Sep 09 20:37:25.312466 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:25.312435    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="380051bc-7985-46d5-9582-51ac0f89c556" path="/var/lib/kubelet/pods/380051bc-7985-46d5-9582-51ac0f89c556/volumes"
Sep 09 20:37:25.372914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:25.372875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:26.092702 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:26.092661    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:26.891549 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:26.891510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:27.591342 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:27.591307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:28.224746 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.224705    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:28.326444 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.326412    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:28.744707 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.744671    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"]
Sep 09 20:37:28.745189 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.745166    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380051bc-7985-46d5-9582-51ac0f89c556" containerName="maas-api"
Sep 09 20:37:28.745265 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.745193    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="380051bc-7985-46d5-9582-51ac0f89c556" containerName="maas-api"
Sep 09 20:37:28.745322 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.745297    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="380051bc-7985-46d5-9582-51ac0f89c556" containerName="maas-api"
Sep 09 20:37:28.748957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.748937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.752250 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.752221    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4101f2\"/\"default-dockercfg-pmn8m\""
Sep 09 20:37:28.752402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.752221    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4101f2\"/\"openshift-service-ca.crt\""
Sep 09 20:37:28.752402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.752283    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4101f2\"/\"kube-root-ca.crt\""
Sep 09 20:37:28.752402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.752376    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4101f2\"/\"auth-test-model-4101f2-kserve-self-signed-certs\""
Sep 09 20:37:28.758335 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.758305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"]
Sep 09 20:37:28.826695 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.826661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca1454a-f8f2-4046-90e1-f8d441de5fa4-tls-certs\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.826695 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.826698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57w2f\" (UniqueName: \"kubernetes.io/projected/cca1454a-f8f2-4046-90e1-f8d441de5fa4-kube-api-access-57w2f\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.826919 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.826761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-model-cache\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.826919 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.826812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-tmp-dir\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.826919 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.826838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-home\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.827036 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.826945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-dshm\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.850420 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.850376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:28.927685 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.927649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca1454a-f8f2-4046-90e1-f8d441de5fa4-tls-certs\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.927685 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.927686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57w2f\" (UniqueName: \"kubernetes.io/projected/cca1454a-f8f2-4046-90e1-f8d441de5fa4-kube-api-access-57w2f\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.927952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.927714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-model-cache\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.927952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.927739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-tmp-dir\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.927952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.927757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-home\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.927952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.927818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-dshm\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.928175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.928138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-model-cache\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.928218 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.928178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-tmp-dir\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.928218 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.928201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-home\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.930085 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.930054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-dshm\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.930219 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.930202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca1454a-f8f2-4046-90e1-f8d441de5fa4-tls-certs\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:28.935414 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:28.935396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57w2f\" (UniqueName: \"kubernetes.io/projected/cca1454a-f8f2-4046-90e1-f8d441de5fa4-kube-api-access-57w2f\") pod \"auth-test-model-4101f2-kserve-6856767499-m26qc\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:29.058899 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:29.058803    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:29.336782 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:29.336705    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"]
Sep 09 20:37:29.340395 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:37:29.340368    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcca1454a_f8f2_4046_90e1_f8d441de5fa4.slice/crio-b8c3528c6a717a71130b148926e42a9a38efe02789945691f2ff3ec2fcf3bc64 WatchSource:0}: Error finding container b8c3528c6a717a71130b148926e42a9a38efe02789945691f2ff3ec2fcf3bc64: Status 404 returned error can't find the container with id b8c3528c6a717a71130b148926e42a9a38efe02789945691f2ff3ec2fcf3bc64
Sep 09 20:37:29.990781 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:29.990745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc" event={"ID":"cca1454a-f8f2-4046-90e1-f8d441de5fa4","Type":"ContainerStarted","Data":"c1fae57c7954c29c3e8110618d3e597cc69840490dd8accfe07276de82d0b254"}
Sep 09 20:37:29.991201 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:29.990789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc" event={"ID":"cca1454a-f8f2-4046-90e1-f8d441de5fa4","Type":"ContainerStarted","Data":"b8c3528c6a717a71130b148926e42a9a38efe02789945691f2ff3ec2fcf3bc64"}
Sep 09 20:37:30.006791 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:30.006742    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc" podStartSLOduration=2.006727477 podStartE2EDuration="2.006727477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:30.005652975 +0000 UTC m=+1257.244755277" watchObservedRunningTime="2026-09-09 20:37:30.006727477 +0000 UTC m=+1257.245829776"
Sep 09 20:37:33.300205 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:33.300170    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:37:33.346436 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:33.346395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:39.059884 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:39.059841    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:39.060348 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:39.059901    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:39.072850 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:39.072822    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:40.053275 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:40.053243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:37:48.602782 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:37:48.602748    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:38:04.995719 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:04.995681    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:38:10.049791 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:10.049751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:10.626370 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:10.626316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:16.050172 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:16.050116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:16.116156 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:16.116116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:16.293041 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:16.293012    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"]
Sep 09 20:38:16.293279 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:16.293255    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerName="istio-proxy" containerID="cri-o://a35389844cb07c75f86024c01b844c231d9ed991951e267f9567f80e60ffbc51" gracePeriod=30
Sep 09 20:38:16.316305 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:16.316222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:17.597279 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:17.597239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:17.698505 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:17.698463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:17.814471 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:17.814436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:18.807892 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:18.807860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:18.908125 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:18.908090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:19.947712 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:19.947674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:21.541310 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.541284    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:38:21.611870 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.611782    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwqmv\" (UniqueName: \"kubernetes.io/projected/460d3a78-6e5f-4411-ab2f-190ac3613348-kube-api-access-jwqmv\") pod \"460d3a78-6e5f-4411-ab2f-190ac3613348\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") "
Sep 09 20:38:21.611870 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.611826    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-data\") pod \"460d3a78-6e5f-4411-ab2f-190ac3613348\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") "
Sep 09 20:38:21.611870 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.611858    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-workload-socket\") pod \"460d3a78-6e5f-4411-ab2f-190ac3613348\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") "
Sep 09 20:38:21.612141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.611893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-credential-socket\") pod \"460d3a78-6e5f-4411-ab2f-190ac3613348\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") "
Sep 09 20:38:21.612141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.611921    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-podinfo\") pod \"460d3a78-6e5f-4411-ab2f-190ac3613348\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") "
Sep 09 20:38:21.612141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.611969    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-token\") pod \"460d3a78-6e5f-4411-ab2f-190ac3613348\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") "
Sep 09 20:38:21.612141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.612023    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460d3a78-6e5f-4411-ab2f-190ac3613348-istiod-ca-cert\") pod \"460d3a78-6e5f-4411-ab2f-190ac3613348\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") "
Sep 09 20:38:21.612141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.612055    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-workload-certs\") pod \"460d3a78-6e5f-4411-ab2f-190ac3613348\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") "
Sep 09 20:38:21.612141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.612084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-envoy\") pod \"460d3a78-6e5f-4411-ab2f-190ac3613348\" (UID: \"460d3a78-6e5f-4411-ab2f-190ac3613348\") "
Sep 09 20:38:21.612449 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.612118    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-data" (OuterVolumeSpecName: "istio-data") pod "460d3a78-6e5f-4411-ab2f-190ac3613348" (UID: "460d3a78-6e5f-4411-ab2f-190ac3613348"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:21.612449 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.612204    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "460d3a78-6e5f-4411-ab2f-190ac3613348" (UID: "460d3a78-6e5f-4411-ab2f-190ac3613348"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:21.612449 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.612218    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "460d3a78-6e5f-4411-ab2f-190ac3613348" (UID: "460d3a78-6e5f-4411-ab2f-190ac3613348"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:21.612449 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.612387    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "460d3a78-6e5f-4411-ab2f-190ac3613348" (UID: "460d3a78-6e5f-4411-ab2f-190ac3613348"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:21.612705 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.612486    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460d3a78-6e5f-4411-ab2f-190ac3613348-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "460d3a78-6e5f-4411-ab2f-190ac3613348" (UID: "460d3a78-6e5f-4411-ab2f-190ac3613348"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:38:21.612705 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.612492    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:38:21.612705 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.612535    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:38:21.612705 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.612552    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:38:21.612705 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.612568    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:38:21.614021 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.613991    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460d3a78-6e5f-4411-ab2f-190ac3613348-kube-api-access-jwqmv" (OuterVolumeSpecName: "kube-api-access-jwqmv") pod "460d3a78-6e5f-4411-ab2f-190ac3613348" (UID: "460d3a78-6e5f-4411-ab2f-190ac3613348"). InnerVolumeSpecName "kube-api-access-jwqmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:38:21.614021 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.614001    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "460d3a78-6e5f-4411-ab2f-190ac3613348" (UID: "460d3a78-6e5f-4411-ab2f-190ac3613348"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:38:21.614184 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.614160    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "460d3a78-6e5f-4411-ab2f-190ac3613348" (UID: "460d3a78-6e5f-4411-ab2f-190ac3613348"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:21.614235 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.614219    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-token" (OuterVolumeSpecName: "istio-token") pod "460d3a78-6e5f-4411-ab2f-190ac3613348" (UID: "460d3a78-6e5f-4411-ab2f-190ac3613348"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:38:21.713665 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.713629    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460d3a78-6e5f-4411-ab2f-190ac3613348-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:38:21.713665 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.713661    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:38:21.713665 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.713670    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwqmv\" (UniqueName: \"kubernetes.io/projected/460d3a78-6e5f-4411-ab2f-190ac3613348-kube-api-access-jwqmv\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:38:21.713958 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.713680    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:38:21.713958 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:21.713688    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460d3a78-6e5f-4411-ab2f-190ac3613348-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:38:22.241785 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:22.241751    2563 generic.go:356] "Generic (PLEG): container finished" podID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerID="a35389844cb07c75f86024c01b844c231d9ed991951e267f9567f80e60ffbc51" exitCode=0
Sep 09 20:38:22.241944 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:22.241801    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"
Sep 09 20:38:22.241944 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:22.241806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" event={"ID":"460d3a78-6e5f-4411-ab2f-190ac3613348","Type":"ContainerDied","Data":"a35389844cb07c75f86024c01b844c231d9ed991951e267f9567f80e60ffbc51"}
Sep 09 20:38:22.241944 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:22.241861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr" event={"ID":"460d3a78-6e5f-4411-ab2f-190ac3613348","Type":"ContainerDied","Data":"462dc6765912c05e504c4d93afdce4f2c7704be3137492b6dff7892aa6b70830"}
Sep 09 20:38:22.241944 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:22.241893    2563 scope.go:117] "RemoveContainer" containerID="a35389844cb07c75f86024c01b844c231d9ed991951e267f9567f80e60ffbc51"
Sep 09 20:38:22.251832 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:22.251807    2563 scope.go:117] "RemoveContainer" containerID="4992638228d40527dab9d6d7ae3c18471e6a0b1f9ec56ff289c3285307aadfa4"
Sep 09 20:38:22.259852 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:22.259829    2563 scope.go:117] "RemoveContainer" containerID="a35389844cb07c75f86024c01b844c231d9ed991951e267f9567f80e60ffbc51"
Sep 09 20:38:22.260106 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:38:22.260090    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35389844cb07c75f86024c01b844c231d9ed991951e267f9567f80e60ffbc51\": container with ID starting with a35389844cb07c75f86024c01b844c231d9ed991951e267f9567f80e60ffbc51 not found: ID does not exist" containerID="a35389844cb07c75f86024c01b844c231d9ed991951e267f9567f80e60ffbc51"
Sep 09 20:38:22.260153 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:22.260113    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35389844cb07c75f86024c01b844c231d9ed991951e267f9567f80e60ffbc51"} err="failed to get container status \"a35389844cb07c75f86024c01b844c231d9ed991951e267f9567f80e60ffbc51\": rpc error: code = NotFound desc = could not find container \"a35389844cb07c75f86024c01b844c231d9ed991951e267f9567f80e60ffbc51\": container with ID starting with a35389844cb07c75f86024c01b844c231d9ed991951e267f9567f80e60ffbc51 not found: ID does not exist"
Sep 09 20:38:22.260153 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:22.260129    2563 scope.go:117] "RemoveContainer" containerID="4992638228d40527dab9d6d7ae3c18471e6a0b1f9ec56ff289c3285307aadfa4"
Sep 09 20:38:22.260405 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:38:22.260387    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4992638228d40527dab9d6d7ae3c18471e6a0b1f9ec56ff289c3285307aadfa4\": container with ID starting with 4992638228d40527dab9d6d7ae3c18471e6a0b1f9ec56ff289c3285307aadfa4 not found: ID does not exist" containerID="4992638228d40527dab9d6d7ae3c18471e6a0b1f9ec56ff289c3285307aadfa4"
Sep 09 20:38:22.260446 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:22.260412    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4992638228d40527dab9d6d7ae3c18471e6a0b1f9ec56ff289c3285307aadfa4"} err="failed to get container status \"4992638228d40527dab9d6d7ae3c18471e6a0b1f9ec56ff289c3285307aadfa4\": rpc error: code = NotFound desc = could not find container \"4992638228d40527dab9d6d7ae3c18471e6a0b1f9ec56ff289c3285307aadfa4\": container with ID starting with 4992638228d40527dab9d6d7ae3c18471e6a0b1f9ec56ff289c3285307aadfa4 not found: ID does not exist"
Sep 09 20:38:22.265832 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:22.265810    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"]
Sep 09 20:38:22.272129 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:22.272105    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-57f5cbb5-openshift-default-56cc4f979b-g4knr"]
Sep 09 20:38:23.313103 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:23.313071    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" path="/var/lib/kubelet/pods/460d3a78-6e5f-4411-ab2f-190ac3613348/volumes"
Sep 09 20:38:23.403243 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:23.403205    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:38:25.598337 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:25.598299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:25.674063 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:25.674030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:26.297155 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:26.297119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:32.268120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:32.268076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:32.334378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:32.334340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:32.413925 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:32.413891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:32.512110 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:32.512071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:33.389052 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:33.389017    2563 scope.go:117] "RemoveContainer" containerID="8062778a5e7b2a8e0b6bad8416cb19ec31c88f0ffa39a45cbbe7058224d49b13"
Sep 09 20:38:33.397236 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:33.397208    2563 scope.go:117] "RemoveContainer" containerID="9c2759074a4a0ea73dac9266b5235b3d1648e4c24948830c22c0fba7270562a0"
Sep 09 20:38:39.703548 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:39.703511    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:38:43.018344 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:43.018306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:43.275711 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:43.275593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:54.600721 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:38:54.600682    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:39:04.275704 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:04.275616    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:09.806608 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:09.806552    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:39:10.226035 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.226000    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:10.326254 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.326201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:10.595263 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.595164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:10.884605 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.884506    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"]
Sep 09 20:39:10.884948 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.884858    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerName="istio-proxy"
Sep 09 20:39:10.884948 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.884870    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerName="istio-proxy"
Sep 09 20:39:10.884948 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.884883    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerName="istio-proxy"
Sep 09 20:39:10.884948 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.884888    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerName="istio-proxy"
Sep 09 20:39:10.885085 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.884956    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerName="istio-proxy"
Sep 09 20:39:10.885085 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.884965    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="460d3a78-6e5f-4411-ab2f-190ac3613348" containerName="istio-proxy"
Sep 09 20:39:10.887857 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.887832    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:10.890253 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.890229    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-077e89-openshift-default-dockercfg-nb6gz\""
Sep 09 20:39:10.899667 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.899640    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"]
Sep 09 20:39:10.907411 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.907381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:10.942950 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.942907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-istio-data\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:10.943120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.942978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-credential-socket\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:10.943120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.943018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013def74-2b21-49ef-be87-f035f8ba5487-istiod-ca-cert\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:10.943120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.943049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013def74-2b21-49ef-be87-f035f8ba5487-istio-token\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:10.943243 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.943165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-workload-certs\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:10.943243 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.943202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-istio-envoy\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:10.943243 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.943236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-workload-socket\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:10.943367 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.943298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013def74-2b21-49ef-be87-f035f8ba5487-istio-podinfo\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:10.943367 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:10.943334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzbtc\" (UniqueName: \"kubernetes.io/projected/013def74-2b21-49ef-be87-f035f8ba5487-kube-api-access-tzbtc\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.044444 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.044386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-credential-socket\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.044669 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.044461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013def74-2b21-49ef-be87-f035f8ba5487-istiod-ca-cert\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.044669 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.044506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013def74-2b21-49ef-be87-f035f8ba5487-istio-token\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.044669 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.044567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-workload-certs\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.044669 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.044608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-istio-envoy\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.044669 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.044636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-workload-socket\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.044991 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.044836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013def74-2b21-49ef-be87-f035f8ba5487-istio-podinfo\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.044991 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.044866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzbtc\" (UniqueName: \"kubernetes.io/projected/013def74-2b21-49ef-be87-f035f8ba5487-kube-api-access-tzbtc\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.045284 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.045256    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013def74-2b21-49ef-be87-f035f8ba5487-istiod-ca-cert\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.045459 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.044956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-workload-certs\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.045599 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.045286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-istio-data\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.045713 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.044907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-credential-socket\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.045713 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.045453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-workload-socket\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.045713 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.045517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-istio-data\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.047560 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.047537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-istio-envoy\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.048014 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.047990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013def74-2b21-49ef-be87-f035f8ba5487-istio-podinfo\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.052938 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.052911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013def74-2b21-49ef-be87-f035f8ba5487-istio-token\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.053450 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.053428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzbtc\" (UniqueName: \"kubernetes.io/projected/013def74-2b21-49ef-be87-f035f8ba5487-kube-api-access-tzbtc\") pod \"e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") " pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.198941 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.198841    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:11.471808 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.471775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:11.482286 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.482264    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"]
Sep 09 20:39:11.484771 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:39:11.484742    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013def74_2b21_49ef_be87_f035f8ba5487.slice/crio-f953135c9a647efb11e7af4d5521d9035a29ebef26feb3d69a651a80681553a5 WatchSource:0}: Error finding container f953135c9a647efb11e7af4d5521d9035a29ebef26feb3d69a651a80681553a5: Status 404 returned error can't find the container with id f953135c9a647efb11e7af4d5521d9035a29ebef26feb3d69a651a80681553a5
Sep 09 20:39:11.487069 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.487030    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:39:11.487160 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.487144    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:39:11.487208 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:11.487188    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:39:12.474718 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:12.474684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" event={"ID":"013def74-2b21-49ef-be87-f035f8ba5487","Type":"ContainerStarted","Data":"c62cd75cfd756be01f5a1a41a4d3c7856370ca4c9b8aecf138956bb233d48a75"}
Sep 09 20:39:12.475123 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:12.474737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" event={"ID":"013def74-2b21-49ef-be87-f035f8ba5487","Type":"ContainerStarted","Data":"f953135c9a647efb11e7af4d5521d9035a29ebef26feb3d69a651a80681553a5"}
Sep 09 20:39:12.499665 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:12.499601    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" podStartSLOduration=2.499583839 podStartE2EDuration="2.499583839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:12.497110117 +0000 UTC m=+1359.736212417" watchObservedRunningTime="2026-09-09 20:39:12.499583839 +0000 UTC m=+1359.738686130"
Sep 09 20:39:13.199704 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:13.199650    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:13.205024 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:13.204998    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:13.479025 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:13.478996    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:13.479883 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:13.479861    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:13.743827 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:13.743731    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:14.420153 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:14.420119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:14.517063 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:14.517022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:14.712363 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:14.712334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:15.476759 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:15.476724    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:16.279118 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:16.279083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:16.950868 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:16.950824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:17.567940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.567886    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"]
Sep 09 20:39:17.572016 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.571991    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:17.574650 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.574620    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-077e89-serving-cert\""
Sep 09 20:39:17.583318 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.583287    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"]
Sep 09 20:39:17.599141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.599107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:17.699236 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.699194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:17.712331 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.712300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b72b963-3eb4-47df-9003-bb022a3a1fbd-openshift-service-ca\") pod \"maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:17.712457 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.712344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-maas-api-tls\") pod \"maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:17.712519 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.712481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-metrics-certs\") pod \"maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:17.712604 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.712564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrkjg\" (UniqueName: \"kubernetes.io/projected/1b72b963-3eb4-47df-9003-bb022a3a1fbd-kube-api-access-rrkjg\") pod \"maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:17.813599 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.813538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrkjg\" (UniqueName: \"kubernetes.io/projected/1b72b963-3eb4-47df-9003-bb022a3a1fbd-kube-api-access-rrkjg\") pod \"maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:17.813809 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.813643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b72b963-3eb4-47df-9003-bb022a3a1fbd-openshift-service-ca\") pod \"maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:17.813809 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.813674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-maas-api-tls\") pod \"maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:17.813809 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.813739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-metrics-certs\") pod \"maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:17.813988 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:39:17.813861    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-serving-cert: secret "maas-api-e2e-api-a-077e89-serving-cert" not found
Sep 09 20:39:17.813988 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:39:17.813953    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-maas-api-tls podName:1b72b963-3eb4-47df-9003-bb022a3a1fbd nodeName:}" failed. No retries permitted until 2026-09-09 20:39:18.313927957 +0000 UTC m=+1365.553030239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-maas-api-tls") pod "maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q" (UID: "1b72b963-3eb4-47df-9003-bb022a3a1fbd") : secret "maas-api-e2e-api-a-077e89-serving-cert" not found
Sep 09 20:39:17.814403 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.814383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b72b963-3eb4-47df-9003-bb022a3a1fbd-openshift-service-ca\") pod \"maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:17.816135 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.816116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-metrics-certs\") pod \"maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:17.824497 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.824427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrkjg\" (UniqueName: \"kubernetes.io/projected/1b72b963-3eb4-47df-9003-bb022a3a1fbd-kube-api-access-rrkjg\") pod \"maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:17.931980 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:17.931946    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:18.068190 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:18.068158    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:18.166938 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:18.166837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:18.319198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:18.319158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-maas-api-tls\") pod \"maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:18.321645 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:18.321624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-maas-api-tls\") pod \"maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:18.485101 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:18.485066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:18.758680 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:18.758510    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"]
Sep 09 20:39:18.761063 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:39:18.761035    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b72b963_3eb4_47df_9003_bb022a3a1fbd.slice/crio-c57f521de6a6a9475f22d2e9875987e9659e12742eeae1ed7c4c0d00ab413d2f WatchSource:0}: Error finding container c57f521de6a6a9475f22d2e9875987e9659e12742eeae1ed7c4c0d00ab413d2f: Status 404 returned error can't find the container with id c57f521de6a6a9475f22d2e9875987e9659e12742eeae1ed7c4c0d00ab413d2f
Sep 09 20:39:19.511035 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:19.511004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q" event={"ID":"1b72b963-3eb4-47df-9003-bb022a3a1fbd","Type":"ContainerStarted","Data":"e489b0f522707756ac3701a054eb806f0de49f85d6ecf6ece2eb920ac807ff5e"}
Sep 09 20:39:19.511210 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:19.511055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q" event={"ID":"1b72b963-3eb4-47df-9003-bb022a3a1fbd","Type":"ContainerStarted","Data":"c57f521de6a6a9475f22d2e9875987e9659e12742eeae1ed7c4c0d00ab413d2f"}
Sep 09 20:39:19.511210 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:19.511099    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:19.529918 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:19.529861    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q" podStartSLOduration=1.9983709219999999 podStartE2EDuration="2.529843361s" totalImagesPullingTime="531.472439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:19.527516422 +0000 UTC m=+1366.766618721" watchObservedRunningTime="2026-09-09 20:39:19.529843361 +0000 UTC m=+1366.768945661"
Sep 09 20:39:20.581885 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:20.581848    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7580-4a9aac"]
Sep 09 20:39:20.585051 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:20.585032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7580-4a9aac"
Sep 09 20:39:20.587366 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:20.587344    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5fctx\""
Sep 09 20:39:20.591261 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:20.591237    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7580-4a9aac"]
Sep 09 20:39:20.641467 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:20.641430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ljt\" (UniqueName: \"kubernetes.io/projected/2b6ccbc0-8a72-4b41-8a9e-18127f32c816-kube-api-access-t4ljt\") pod \"test-curl-7580-4a9aac\" (UID: \"2b6ccbc0-8a72-4b41-8a9e-18127f32c816\") " pod="opendatahub/test-curl-7580-4a9aac"
Sep 09 20:39:20.742545 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:20.742504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ljt\" (UniqueName: \"kubernetes.io/projected/2b6ccbc0-8a72-4b41-8a9e-18127f32c816-kube-api-access-t4ljt\") pod \"test-curl-7580-4a9aac\" (UID: \"2b6ccbc0-8a72-4b41-8a9e-18127f32c816\") " pod="opendatahub/test-curl-7580-4a9aac"
Sep 09 20:39:20.750331 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:20.750300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ljt\" (UniqueName: \"kubernetes.io/projected/2b6ccbc0-8a72-4b41-8a9e-18127f32c816-kube-api-access-t4ljt\") pod \"test-curl-7580-4a9aac\" (UID: \"2b6ccbc0-8a72-4b41-8a9e-18127f32c816\") " pod="opendatahub/test-curl-7580-4a9aac"
Sep 09 20:39:20.895611 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:20.895497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7580-4a9aac"
Sep 09 20:39:21.165311 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:21.165285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7580-4a9aac"]
Sep 09 20:39:21.167813 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:39:21.167779    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b6ccbc0_8a72_4b41_8a9e_18127f32c816.slice/crio-6e718fd5fd878608184718df15e4de3f4cb4df5724a3d25054b295c318b889e7 WatchSource:0}: Error finding container 6e718fd5fd878608184718df15e4de3f4cb4df5724a3d25054b295c318b889e7: Status 404 returned error can't find the container with id 6e718fd5fd878608184718df15e4de3f4cb4df5724a3d25054b295c318b889e7
Sep 09 20:39:21.529150 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:21.529106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7580-4a9aac" event={"ID":"2b6ccbc0-8a72-4b41-8a9e-18127f32c816","Type":"ContainerStarted","Data":"22e8eddde984800862e93d0815beea1ea2d4a71b9ca7ebc6c2158963790cbb32"}
Sep 09 20:39:21.529150 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:21.529153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7580-4a9aac" event={"ID":"2b6ccbc0-8a72-4b41-8a9e-18127f32c816","Type":"ContainerStarted","Data":"6e718fd5fd878608184718df15e4de3f4cb4df5724a3d25054b295c318b889e7"}
Sep 09 20:39:21.562205 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:21.562145    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7580-4a9aac" podStartSLOduration=1.562128302 podStartE2EDuration="1.562128302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:21.560255245 +0000 UTC m=+1368.799357550" watchObservedRunningTime="2026-09-09 20:39:21.562128302 +0000 UTC m=+1368.801230602"
Sep 09 20:39:22.149320 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:22.149283    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7580-4a9aac"]
Sep 09 20:39:22.155346 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:22.155315    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7580-4a9aac"]
Sep 09 20:39:22.534111 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:39:22.534079    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7580-4a9aac" container="test-curl-7580-4a9aac"
Sep 09 20:39:23.487644 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:23.487600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:23.538985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:23.538946    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7580-4a9aac" podUID="2b6ccbc0-8a72-4b41-8a9e-18127f32c816" containerName="test-curl-7580-4a9aac" containerID="cri-o://22e8eddde984800862e93d0815beea1ea2d4a71b9ca7ebc6c2158963790cbb32" gracePeriod=2
Sep 09 20:39:24.354027 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:24.353989    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"]
Sep 09 20:39:24.378242 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:24.378201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:25.522118 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:25.522091    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:39:25.708056 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:25.708025    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7580-4a9aac"
Sep 09 20:39:25.792280 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:25.792190    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4ljt\" (UniqueName: \"kubernetes.io/projected/2b6ccbc0-8a72-4b41-8a9e-18127f32c816-kube-api-access-t4ljt\") pod \"2b6ccbc0-8a72-4b41-8a9e-18127f32c816\" (UID: \"2b6ccbc0-8a72-4b41-8a9e-18127f32c816\") "
Sep 09 20:39:25.794322 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:25.794290    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6ccbc0-8a72-4b41-8a9e-18127f32c816-kube-api-access-t4ljt" (OuterVolumeSpecName: "kube-api-access-t4ljt") pod "2b6ccbc0-8a72-4b41-8a9e-18127f32c816" (UID: "2b6ccbc0-8a72-4b41-8a9e-18127f32c816"). InnerVolumeSpecName "kube-api-access-t4ljt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:39:25.894019 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:25.893969    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4ljt\" (UniqueName: \"kubernetes.io/projected/2b6ccbc0-8a72-4b41-8a9e-18127f32c816-kube-api-access-t4ljt\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:39:26.204165 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:26.204062    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:39:26.556513 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:26.556473    2563 generic.go:356] "Generic (PLEG): container finished" podID="2b6ccbc0-8a72-4b41-8a9e-18127f32c816" containerID="22e8eddde984800862e93d0815beea1ea2d4a71b9ca7ebc6c2158963790cbb32" exitCode=137
Sep 09 20:39:26.556952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:26.556539    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7580-4a9aac"
Sep 09 20:39:26.556952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:26.556545    2563 scope.go:117] "RemoveContainer" containerID="22e8eddde984800862e93d0815beea1ea2d4a71b9ca7ebc6c2158963790cbb32"
Sep 09 20:39:26.565223 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:26.565204    2563 scope.go:117] "RemoveContainer" containerID="22e8eddde984800862e93d0815beea1ea2d4a71b9ca7ebc6c2158963790cbb32"
Sep 09 20:39:26.565512 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:39:26.565490    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e8eddde984800862e93d0815beea1ea2d4a71b9ca7ebc6c2158963790cbb32\": container with ID starting with 22e8eddde984800862e93d0815beea1ea2d4a71b9ca7ebc6c2158963790cbb32 not found: ID does not exist" containerID="22e8eddde984800862e93d0815beea1ea2d4a71b9ca7ebc6c2158963790cbb32"
Sep 09 20:39:26.565550 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:26.565522    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e8eddde984800862e93d0815beea1ea2d4a71b9ca7ebc6c2158963790cbb32"} err="failed to get container status \"22e8eddde984800862e93d0815beea1ea2d4a71b9ca7ebc6c2158963790cbb32\": rpc error: code = NotFound desc = could not find container \"22e8eddde984800862e93d0815beea1ea2d4a71b9ca7ebc6c2158963790cbb32\": container with ID starting with 22e8eddde984800862e93d0815beea1ea2d4a71b9ca7ebc6c2158963790cbb32 not found: ID does not exist"
Sep 09 20:39:26.572038 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:39:26.572010    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7580-4a9aac\" is forbidden: User \"system:node:ip-10-0-143-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-241.ec2.internal' and this object" podUID="2b6ccbc0-8a72-4b41-8a9e-18127f32c816" pod="opendatahub/test-curl-7580-4a9aac"
Sep 09 20:39:27.312914 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:27.312867    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b6ccbc0-8a72-4b41-8a9e-18127f32c816" path="/var/lib/kubelet/pods/2b6ccbc0-8a72-4b41-8a9e-18127f32c816/volumes"
Sep 09 20:39:35.830560 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.830519    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"]
Sep 09 20:39:35.831171 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.831053    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6ccbc0-8a72-4b41-8a9e-18127f32c816" containerName="test-curl-7580-4a9aac"
Sep 09 20:39:35.831171 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.831074    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6ccbc0-8a72-4b41-8a9e-18127f32c816" containerName="test-curl-7580-4a9aac"
Sep 09 20:39:35.831337 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.831181    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6ccbc0-8a72-4b41-8a9e-18127f32c816" containerName="test-curl-7580-4a9aac"
Sep 09 20:39:35.835131 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.835108    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:35.837592 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.837546    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-166b94\"/\"kube-root-ca.crt\""
Sep 09 20:39:35.838683 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.838663    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-166b94\"/\"openshift-service-ca.crt\""
Sep 09 20:39:35.838794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.838727    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-166b94\"/\"default-dockercfg-9zkh8\""
Sep 09 20:39:35.838794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.838778    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-166b94\"/\"sub-test-model-166b94-kserve-self-signed-certs\""
Sep 09 20:39:35.842379 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.842349    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"]
Sep 09 20:39:35.857190 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.857153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:35.990160 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.990113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-home\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:35.990358 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.990189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-dshm\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:35.990358 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.990212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc38f83d-11f6-45c3-9c69-b0504ce6be58-tls-certs\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:35.990358 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.990258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc24z\" (UniqueName: \"kubernetes.io/projected/cc38f83d-11f6-45c3-9c69-b0504ce6be58-kube-api-access-tc24z\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:35.990358 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.990284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-tmp-dir\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:35.990358 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:35.990316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-model-cache\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.091696 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.091565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc24z\" (UniqueName: \"kubernetes.io/projected/cc38f83d-11f6-45c3-9c69-b0504ce6be58-kube-api-access-tc24z\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.091879 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.091734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-tmp-dir\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.091879 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.091783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-model-cache\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.092010 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.091884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-home\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.092010 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.091947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-dshm\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.092010 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.091970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc38f83d-11f6-45c3-9c69-b0504ce6be58-tls-certs\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.092136 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.092119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-tmp-dir\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.092201 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.092182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-model-cache\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.092261 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.092240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-home\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.094236 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.094201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-dshm\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.094410 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.094392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc38f83d-11f6-45c3-9c69-b0504ce6be58-tls-certs\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.099704 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.099680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc24z\" (UniqueName: \"kubernetes.io/projected/cc38f83d-11f6-45c3-9c69-b0504ce6be58-kube-api-access-tc24z\") pod \"sub-test-model-166b94-kserve-6c75547595-d24nk\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") " pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.146605 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.146551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:36.430472 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.430444    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"]
Sep 09 20:39:36.432404 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:39:36.432380    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc38f83d_11f6_45c3_9c69_b0504ce6be58.slice/crio-72ca09074b4fc0f0c3c7eefaef1a1f9b51bc7139eb8e9457f86cbcb36e768584 WatchSource:0}: Error finding container 72ca09074b4fc0f0c3c7eefaef1a1f9b51bc7139eb8e9457f86cbcb36e768584: Status 404 returned error can't find the container with id 72ca09074b4fc0f0c3c7eefaef1a1f9b51bc7139eb8e9457f86cbcb36e768584
Sep 09 20:39:36.609797 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.609756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk" event={"ID":"cc38f83d-11f6-45c3-9c69-b0504ce6be58","Type":"ContainerStarted","Data":"26847d4187e682a8cc5660e58df8d5df9e807d066bcd9b98f52db70820a78239"}
Sep 09 20:39:36.610003 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.609812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk" event={"ID":"cc38f83d-11f6-45c3-9c69-b0504ce6be58","Type":"ContainerStarted","Data":"72ca09074b4fc0f0c3c7eefaef1a1f9b51bc7139eb8e9457f86cbcb36e768584"}
Sep 09 20:39:36.611394 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.611369    2563 generic.go:356] "Generic (PLEG): container finished" podID="013def74-2b21-49ef-be87-f035f8ba5487" containerID="c62cd75cfd756be01f5a1a41a4d3c7856370ca4c9b8aecf138956bb233d48a75" exitCode=0
Sep 09 20:39:36.611514 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.611440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" event={"ID":"013def74-2b21-49ef-be87-f035f8ba5487","Type":"ContainerDied","Data":"c62cd75cfd756be01f5a1a41a4d3c7856370ca4c9b8aecf138956bb233d48a75"}
Sep 09 20:39:36.611784 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.611771    2563 scope.go:117] "RemoveContainer" containerID="c62cd75cfd756be01f5a1a41a4d3c7856370ca4c9b8aecf138956bb233d48a75"
Sep 09 20:39:36.612816 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.612786    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:39:36.612924 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.612848    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:39:36.612924 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.612882    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:39:36.626210 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:36.626169    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk" podStartSLOduration=1.626155254 podStartE2EDuration="1.626155254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:36.624813184 +0000 UTC m=+1383.863915478" watchObservedRunningTime="2026-09-09 20:39:36.626155254 +0000 UTC m=+1383.865257553"
Sep 09 20:39:37.618734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:37.618699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" event={"ID":"013def74-2b21-49ef-be87-f035f8ba5487","Type":"ContainerStarted","Data":"aa2235338d06b76c5ac76b32eff35e3e1542fd0e4a4d421cfc269a21aeeff609"}
Sep 09 20:39:38.199459 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:38.199416    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:38.199658 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:38.199625    2563 patch_prober.go:28] interesting pod/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 09 20:39:38.199716 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:38.199680    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" podUID="013def74-2b21-49ef-be87-f035f8ba5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 09 20:39:39.199644 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.199600    2563 patch_prober.go:28] interesting pod/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 09 20:39:39.200087 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.199670    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" podUID="013def74-2b21-49ef-be87-f035f8ba5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 09 20:39:39.368352 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.368312    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:39.376563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.376527    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc" podUID="cca1454a-f8f2-4046-90e1-f8d441de5fa4" containerName="main" containerID="cri-o://c1fae57c7954c29c3e8110618d3e597cc69840490dd8accfe07276de82d0b254" gracePeriod=45
Sep 09 20:39:39.579901 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.579861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:39.629452 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.629421    2563 generic.go:356] "Generic (PLEG): container finished" podID="cca1454a-f8f2-4046-90e1-f8d441de5fa4" containerID="c1fae57c7954c29c3e8110618d3e597cc69840490dd8accfe07276de82d0b254" exitCode=0
Sep 09 20:39:39.629649 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.629492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc" event={"ID":"cca1454a-f8f2-4046-90e1-f8d441de5fa4","Type":"ContainerDied","Data":"c1fae57c7954c29c3e8110618d3e597cc69840490dd8accfe07276de82d0b254"}
Sep 09 20:39:39.677979 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.677946    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:39:39.726051 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.726016    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca1454a-f8f2-4046-90e1-f8d441de5fa4-tls-certs\") pod \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") "
Sep 09 20:39:39.726248 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.726069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-home\") pod \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") "
Sep 09 20:39:39.726248 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.726127    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-tmp-dir\") pod \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") "
Sep 09 20:39:39.726248 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.726167    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-model-cache\") pod \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") "
Sep 09 20:39:39.726248 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.726188    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-dshm\") pod \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") "
Sep 09 20:39:39.726464 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.726256    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57w2f\" (UniqueName: \"kubernetes.io/projected/cca1454a-f8f2-4046-90e1-f8d441de5fa4-kube-api-access-57w2f\") pod \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\" (UID: \"cca1454a-f8f2-4046-90e1-f8d441de5fa4\") "
Sep 09 20:39:39.726464 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.726341    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-home" (OuterVolumeSpecName: "home") pod "cca1454a-f8f2-4046-90e1-f8d441de5fa4" (UID: "cca1454a-f8f2-4046-90e1-f8d441de5fa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:39.726593 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.726450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cca1454a-f8f2-4046-90e1-f8d441de5fa4" (UID: "cca1454a-f8f2-4046-90e1-f8d441de5fa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:39.726660 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.726601    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-model-cache" (OuterVolumeSpecName: "model-cache") pod "cca1454a-f8f2-4046-90e1-f8d441de5fa4" (UID: "cca1454a-f8f2-4046-90e1-f8d441de5fa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:39.726660 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.726625    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-home\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:39:39.726660 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.726644    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-tmp-dir\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:39:39.728325 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.728300    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca1454a-f8f2-4046-90e1-f8d441de5fa4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cca1454a-f8f2-4046-90e1-f8d441de5fa4" (UID: "cca1454a-f8f2-4046-90e1-f8d441de5fa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:39:39.728418 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.728375    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca1454a-f8f2-4046-90e1-f8d441de5fa4-kube-api-access-57w2f" (OuterVolumeSpecName: "kube-api-access-57w2f") pod "cca1454a-f8f2-4046-90e1-f8d441de5fa4" (UID: "cca1454a-f8f2-4046-90e1-f8d441de5fa4"). InnerVolumeSpecName "kube-api-access-57w2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:39:39.728491 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.728471    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-dshm" (OuterVolumeSpecName: "dshm") pod "cca1454a-f8f2-4046-90e1-f8d441de5fa4" (UID: "cca1454a-f8f2-4046-90e1-f8d441de5fa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:39.827195 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.827101    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57w2f\" (UniqueName: \"kubernetes.io/projected/cca1454a-f8f2-4046-90e1-f8d441de5fa4-kube-api-access-57w2f\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:39:39.827195 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.827142    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca1454a-f8f2-4046-90e1-f8d441de5fa4-tls-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:39:39.827195 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.827152    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-model-cache\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:39:39.827195 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.827161    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca1454a-f8f2-4046-90e1-f8d441de5fa4-dshm\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:39:39.836466 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:39.836436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:40.200235 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:40.200138    2563 patch_prober.go:28] interesting pod/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 09 20:39:40.200235 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:40.200198    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" podUID="013def74-2b21-49ef-be87-f035f8ba5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 09 20:39:40.235171 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:40.235131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:40.387358 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:40.387310    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:40.522872 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:40.522836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:40.636120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:40.636088    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"
Sep 09 20:39:40.636120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:40.636115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc" event={"ID":"cca1454a-f8f2-4046-90e1-f8d441de5fa4","Type":"ContainerDied","Data":"b8c3528c6a717a71130b148926e42a9a38efe02789945691f2ff3ec2fcf3bc64"}
Sep 09 20:39:40.636356 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:40.636163    2563 scope.go:117] "RemoveContainer" containerID="c1fae57c7954c29c3e8110618d3e597cc69840490dd8accfe07276de82d0b254"
Sep 09 20:39:40.661645 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:40.661611    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"]
Sep 09 20:39:40.664285 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:40.664253    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/auth-test-model-4101f2-kserve-6856767499-m26qc"]
Sep 09 20:39:41.193728 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:41.193688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:41.199763 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:41.199740    2563 patch_prober.go:28] interesting pod/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 09 20:39:41.199844 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:41.199785    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" podUID="013def74-2b21-49ef-be87-f035f8ba5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 09 20:39:41.314319 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:41.314284    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cca1454a-f8f2-4046-90e1-f8d441de5fa4" path="/var/lib/kubelet/pods/cca1454a-f8f2-4046-90e1-f8d441de5fa4/volumes"
Sep 09 20:39:41.849612 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:41.849558    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:42.200109 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:42.200024    2563 patch_prober.go:28] interesting pod/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 09 20:39:42.200109 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:42.200079    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" podUID="013def74-2b21-49ef-be87-f035f8ba5487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 09 20:39:42.499864 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:42.499827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:43.267293 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:43.267231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:43.403567 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:43.403514    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:43.412210 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:43.412175    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:43.413933 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:43.413904    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:39:43.702611 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:43.702494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:43.800863 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:43.800820    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:44.095623 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:44.095555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:44.741542 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:44.741499    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:45.685065 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:45.685032    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:46.147360 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:46.147317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:46.147901 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:46.147480    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:46.160610 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:46.160558    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:46.331861 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:46.331821    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:46.680264 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:46.680233    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:39:46.797441 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:46.797403    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:39:47.029180 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:47.029143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:47.868841 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:47.868804    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"]
Sep 09 20:39:47.869376 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:47.869355    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca1454a-f8f2-4046-90e1-f8d441de5fa4" containerName="main"
Sep 09 20:39:47.869446 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:47.869382    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca1454a-f8f2-4046-90e1-f8d441de5fa4" containerName="main"
Sep 09 20:39:47.869520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:47.869507    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cca1454a-f8f2-4046-90e1-f8d441de5fa4" containerName="main"
Sep 09 20:39:47.873089 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:47.873065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:47.875270 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:47.875237    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-06f7dc-serving-cert\""
Sep 09 20:39:47.883789 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:47.883763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"]
Sep 09 20:39:47.900324 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:47.900297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-metrics-certs\") pod \"maas-api-e2e-api-b-06f7dc-788d56695f-mttgl\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:47.900507 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:47.900329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tm7f\" (UniqueName: \"kubernetes.io/projected/276b4f79-c28f-4c68-9e20-10cf4e00eca9-kube-api-access-7tm7f\") pod \"maas-api-e2e-api-b-06f7dc-788d56695f-mttgl\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:47.900507 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:47.900397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276b4f79-c28f-4c68-9e20-10cf4e00eca9-openshift-service-ca\") pod \"maas-api-e2e-api-b-06f7dc-788d56695f-mttgl\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:47.900507 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:47.900436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-maas-api-tls\") pod \"maas-api-e2e-api-b-06f7dc-788d56695f-mttgl\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:47.902972 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:47.902948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:48.001289 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.001252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276b4f79-c28f-4c68-9e20-10cf4e00eca9-openshift-service-ca\") pod \"maas-api-e2e-api-b-06f7dc-788d56695f-mttgl\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:48.001289 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.001288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-maas-api-tls\") pod \"maas-api-e2e-api-b-06f7dc-788d56695f-mttgl\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:48.001551 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.001364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-metrics-certs\") pod \"maas-api-e2e-api-b-06f7dc-788d56695f-mttgl\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:48.001551 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.001387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tm7f\" (UniqueName: \"kubernetes.io/projected/276b4f79-c28f-4c68-9e20-10cf4e00eca9-kube-api-access-7tm7f\") pod \"maas-api-e2e-api-b-06f7dc-788d56695f-mttgl\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:48.001551 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:39:48.001466    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-serving-cert: secret "maas-api-e2e-api-b-06f7dc-serving-cert" not found
Sep 09 20:39:48.001551 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:39:48.001537    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-maas-api-tls podName:276b4f79-c28f-4c68-9e20-10cf4e00eca9 nodeName:}" failed. No retries permitted until 2026-09-09 20:39:48.50151954 +0000 UTC m=+1395.740621818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-maas-api-tls") pod "maas-api-e2e-api-b-06f7dc-788d56695f-mttgl" (UID: "276b4f79-c28f-4c68-9e20-10cf4e00eca9") : secret "maas-api-e2e-api-b-06f7dc-serving-cert" not found
Sep 09 20:39:48.002036 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.002013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276b4f79-c28f-4c68-9e20-10cf4e00eca9-openshift-service-ca\") pod \"maas-api-e2e-api-b-06f7dc-788d56695f-mttgl\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:48.004033 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.004009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-metrics-certs\") pod \"maas-api-e2e-api-b-06f7dc-788d56695f-mttgl\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:48.004275 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.004236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:48.009641 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.009619    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tm7f\" (UniqueName: \"kubernetes.io/projected/276b4f79-c28f-4c68-9e20-10cf4e00eca9-kube-api-access-7tm7f\") pod \"maas-api-e2e-api-b-06f7dc-788d56695f-mttgl\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:48.386526 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.386471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:48.491043 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.491009    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:48.507265 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.507231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-maas-api-tls\") pod \"maas-api-e2e-api-b-06f7dc-788d56695f-mttgl\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:48.509641 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.509612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-maas-api-tls\") pod \"maas-api-e2e-api-b-06f7dc-788d56695f-mttgl\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:48.563923 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.563888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:48.786754 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:48.786721    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:49.061796 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:49.061713    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"]
Sep 09 20:39:49.064805 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:39:49.064776    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276b4f79_c28f_4c68_9e20_10cf4e00eca9.slice/crio-f611c169ea10a491c2380c3cb32fdaf9ca1e74d738ca65ec6e7caee8187f90b0 WatchSource:0}: Error finding container f611c169ea10a491c2380c3cb32fdaf9ca1e74d738ca65ec6e7caee8187f90b0: Status 404 returned error can't find the container with id f611c169ea10a491c2380c3cb32fdaf9ca1e74d738ca65ec6e7caee8187f90b0
Sep 09 20:39:49.685527 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:49.685430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl" event={"ID":"276b4f79-c28f-4c68-9e20-10cf4e00eca9","Type":"ContainerStarted","Data":"44c68a1f21bdd9bcef09cab50fe0669186a5e5bd04fd20c3c279b225bad421f6"}
Sep 09 20:39:49.685744 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:49.685527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl" event={"ID":"276b4f79-c28f-4c68-9e20-10cf4e00eca9","Type":"ContainerStarted","Data":"f611c169ea10a491c2380c3cb32fdaf9ca1e74d738ca65ec6e7caee8187f90b0"}
Sep 09 20:39:49.685744 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:49.685559    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:39:49.704125 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:49.704079    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl" podStartSLOduration=2.37770635 podStartE2EDuration="2.704064316s" totalImagesPullingTime="326.357966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:49.702432859 +0000 UTC m=+1396.941535161" watchObservedRunningTime="2026-09-09 20:39:49.704064316 +0000 UTC m=+1396.943166618"
Sep 09 20:39:55.696003 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:39:55.695973    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:40:03.300247 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:03.300208    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:40:10.011182 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:10.011136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:11.075106 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:11.075064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:11.809776 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:11.809740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:12.527877 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:12.527834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:13.391043 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:13.391007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:13.749314 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:13.749266    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:14.204132 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:14.204040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:14.966958 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:14.966918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:16.690701 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:16.690662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:17.537606 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:17.537555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:18.092839 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:18.092802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:18.462624 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:18.462565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:19.105032 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:19.104993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:19.871937 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:19.871900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:20.293232 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:20.293194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:20.395203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:20.395144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:20.552080 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:20.551985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:21.277287 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:21.277246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:22.082295 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:22.082260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:23.707199 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:23.707158    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:40:32.017294 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.017208    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"]
Sep 09 20:40:32.017742 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.017442    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q" podUID="1b72b963-3eb4-47df-9003-bb022a3a1fbd" containerName="maas-api" containerID="cri-o://e489b0f522707756ac3701a054eb806f0de49f85d6ecf6ece2eb920ac807ff5e" gracePeriod=30
Sep 09 20:40:32.042562 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.042526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:32.105955 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.105920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:32.206012 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.205979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:32.268473 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.268412    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:40:32.295725 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.295567    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:32.298004 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.297971    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b72b963-3eb4-47df-9003-bb022a3a1fbd-openshift-service-ca\") pod \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") "
Sep 09 20:40:32.298156 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.298026    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-metrics-certs\") pod \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") "
Sep 09 20:40:32.298156 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.298084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrkjg\" (UniqueName: \"kubernetes.io/projected/1b72b963-3eb4-47df-9003-bb022a3a1fbd-kube-api-access-rrkjg\") pod \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") "
Sep 09 20:40:32.298156 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.298123    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-maas-api-tls\") pod \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\" (UID: \"1b72b963-3eb4-47df-9003-bb022a3a1fbd\") "
Sep 09 20:40:32.299315 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.299278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b72b963-3eb4-47df-9003-bb022a3a1fbd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b72b963-3eb4-47df-9003-bb022a3a1fbd" (UID: "1b72b963-3eb4-47df-9003-bb022a3a1fbd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:32.307792 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.307750    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b72b963-3eb4-47df-9003-bb022a3a1fbd" (UID: "1b72b963-3eb4-47df-9003-bb022a3a1fbd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:40:32.307792 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.307778    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b72b963-3eb4-47df-9003-bb022a3a1fbd-kube-api-access-rrkjg" (OuterVolumeSpecName: "kube-api-access-rrkjg") pod "1b72b963-3eb4-47df-9003-bb022a3a1fbd" (UID: "1b72b963-3eb4-47df-9003-bb022a3a1fbd"). InnerVolumeSpecName "kube-api-access-rrkjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:32.320459 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.320427    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1b72b963-3eb4-47df-9003-bb022a3a1fbd" (UID: "1b72b963-3eb4-47df-9003-bb022a3a1fbd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:40:32.399474 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.399436    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b72b963-3eb4-47df-9003-bb022a3a1fbd-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.399474 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.399470    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.399474 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.399479    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrkjg\" (UniqueName: \"kubernetes.io/projected/1b72b963-3eb4-47df-9003-bb022a3a1fbd-kube-api-access-rrkjg\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.399728 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.399488    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b72b963-3eb4-47df-9003-bb022a3a1fbd-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.624881 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.624788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:32.709957 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.709921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:32.900112 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.900021    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b72b963-3eb4-47df-9003-bb022a3a1fbd" containerID="e489b0f522707756ac3701a054eb806f0de49f85d6ecf6ece2eb920ac807ff5e" exitCode=0
Sep 09 20:40:32.900112 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.900074    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"
Sep 09 20:40:32.900112 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.900105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q" event={"ID":"1b72b963-3eb4-47df-9003-bb022a3a1fbd","Type":"ContainerDied","Data":"e489b0f522707756ac3701a054eb806f0de49f85d6ecf6ece2eb920ac807ff5e"}
Sep 09 20:40:32.900392 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.900156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q" event={"ID":"1b72b963-3eb4-47df-9003-bb022a3a1fbd","Type":"ContainerDied","Data":"c57f521de6a6a9475f22d2e9875987e9659e12742eeae1ed7c4c0d00ab413d2f"}
Sep 09 20:40:32.900392 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.900180    2563 scope.go:117] "RemoveContainer" containerID="e489b0f522707756ac3701a054eb806f0de49f85d6ecf6ece2eb920ac807ff5e"
Sep 09 20:40:32.915901 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.915882    2563 scope.go:117] "RemoveContainer" containerID="e489b0f522707756ac3701a054eb806f0de49f85d6ecf6ece2eb920ac807ff5e"
Sep 09 20:40:32.916184 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:40:32.916158    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e489b0f522707756ac3701a054eb806f0de49f85d6ecf6ece2eb920ac807ff5e\": container with ID starting with e489b0f522707756ac3701a054eb806f0de49f85d6ecf6ece2eb920ac807ff5e not found: ID does not exist" containerID="e489b0f522707756ac3701a054eb806f0de49f85d6ecf6ece2eb920ac807ff5e"
Sep 09 20:40:32.916244 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.916197    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e489b0f522707756ac3701a054eb806f0de49f85d6ecf6ece2eb920ac807ff5e"} err="failed to get container status \"e489b0f522707756ac3701a054eb806f0de49f85d6ecf6ece2eb920ac807ff5e\": rpc error: code = NotFound desc = could not find container \"e489b0f522707756ac3701a054eb806f0de49f85d6ecf6ece2eb920ac807ff5e\": container with ID starting with e489b0f522707756ac3701a054eb806f0de49f85d6ecf6ece2eb920ac807ff5e not found: ID does not exist"
Sep 09 20:40:32.930380 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.930355    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"]
Sep 09 20:40:32.934144 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:32.934119    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-077e89-f5fff9bbd-rb86q"]
Sep 09 20:40:33.312862 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:33.312830    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b72b963-3eb4-47df-9003-bb022a3a1fbd" path="/var/lib/kubelet/pods/1b72b963-3eb4-47df-9003-bb022a3a1fbd/volumes"
Sep 09 20:40:33.461994 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:33.461964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:34.173343 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:34.173302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:34.816665 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:34.816631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:35.470527 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:35.470492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:36.118763 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:36.118727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:38.527060 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:38.527022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:38.620022 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:38.619980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:38.620967 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:38.620947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:40:38.686660 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:38.686627    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"]
Sep 09 20:40:38.686902 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:38.686870    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" podUID="013def74-2b21-49ef-be87-f035f8ba5487" containerName="istio-proxy" containerID="cri-o://aa2235338d06b76c5ac76b32eff35e3e1542fd0e4a4d421cfc269a21aeeff609" gracePeriod=30
Sep 09 20:40:38.711999 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:38.711970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:39.388562 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:39.388528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:40.122703 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:40.122652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:40.883087 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:40.883036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:41.702139 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:41.702100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:42.443220 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:42.443181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:43.091370 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.091329    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:43.414618 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.414513    2563 patch_prober.go:28] interesting pod/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 20:40:43.414618 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.414564    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" podUID="013def74-2b21-49ef-be87-f035f8ba5487" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:40:43.816419 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.816379    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:43.941438 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.941414    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:40:43.957409 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.957379    2563 generic.go:356] "Generic (PLEG): container finished" podID="013def74-2b21-49ef-be87-f035f8ba5487" containerID="aa2235338d06b76c5ac76b32eff35e3e1542fd0e4a4d421cfc269a21aeeff609" exitCode=0
Sep 09 20:40:43.957593 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.957430    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"
Sep 09 20:40:43.957593 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.957464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" event={"ID":"013def74-2b21-49ef-be87-f035f8ba5487","Type":"ContainerDied","Data":"aa2235338d06b76c5ac76b32eff35e3e1542fd0e4a4d421cfc269a21aeeff609"}
Sep 09 20:40:43.957593 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.957548    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs" event={"ID":"013def74-2b21-49ef-be87-f035f8ba5487","Type":"ContainerDied","Data":"f953135c9a647efb11e7af4d5521d9035a29ebef26feb3d69a651a80681553a5"}
Sep 09 20:40:43.957713 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.957596    2563 scope.go:117] "RemoveContainer" containerID="aa2235338d06b76c5ac76b32eff35e3e1542fd0e4a4d421cfc269a21aeeff609"
Sep 09 20:40:43.967522 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.967503    2563 scope.go:117] "RemoveContainer" containerID="c62cd75cfd756be01f5a1a41a4d3c7856370ca4c9b8aecf138956bb233d48a75"
Sep 09 20:40:43.977213 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.977183    2563 scope.go:117] "RemoveContainer" containerID="aa2235338d06b76c5ac76b32eff35e3e1542fd0e4a4d421cfc269a21aeeff609"
Sep 09 20:40:43.977520 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:40:43.977500    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2235338d06b76c5ac76b32eff35e3e1542fd0e4a4d421cfc269a21aeeff609\": container with ID starting with aa2235338d06b76c5ac76b32eff35e3e1542fd0e4a4d421cfc269a21aeeff609 not found: ID does not exist" containerID="aa2235338d06b76c5ac76b32eff35e3e1542fd0e4a4d421cfc269a21aeeff609"
Sep 09 20:40:43.977598 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.977530    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2235338d06b76c5ac76b32eff35e3e1542fd0e4a4d421cfc269a21aeeff609"} err="failed to get container status \"aa2235338d06b76c5ac76b32eff35e3e1542fd0e4a4d421cfc269a21aeeff609\": rpc error: code = NotFound desc = could not find container \"aa2235338d06b76c5ac76b32eff35e3e1542fd0e4a4d421cfc269a21aeeff609\": container with ID starting with aa2235338d06b76c5ac76b32eff35e3e1542fd0e4a4d421cfc269a21aeeff609 not found: ID does not exist"
Sep 09 20:40:43.977598 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.977548    2563 scope.go:117] "RemoveContainer" containerID="c62cd75cfd756be01f5a1a41a4d3c7856370ca4c9b8aecf138956bb233d48a75"
Sep 09 20:40:43.977875 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:40:43.977854    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62cd75cfd756be01f5a1a41a4d3c7856370ca4c9b8aecf138956bb233d48a75\": container with ID starting with c62cd75cfd756be01f5a1a41a4d3c7856370ca4c9b8aecf138956bb233d48a75 not found: ID does not exist" containerID="c62cd75cfd756be01f5a1a41a4d3c7856370ca4c9b8aecf138956bb233d48a75"
Sep 09 20:40:43.977939 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:43.977884    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62cd75cfd756be01f5a1a41a4d3c7856370ca4c9b8aecf138956bb233d48a75"} err="failed to get container status \"c62cd75cfd756be01f5a1a41a4d3c7856370ca4c9b8aecf138956bb233d48a75\": rpc error: code = NotFound desc = could not find container \"c62cd75cfd756be01f5a1a41a4d3c7856370ca4c9b8aecf138956bb233d48a75\": container with ID starting with c62cd75cfd756be01f5a1a41a4d3c7856370ca4c9b8aecf138956bb233d48a75 not found: ID does not exist"
Sep 09 20:40:44.001323 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.001290    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013def74-2b21-49ef-be87-f035f8ba5487-istio-podinfo\") pod \"013def74-2b21-49ef-be87-f035f8ba5487\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") "
Sep 09 20:40:44.001511 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.001343    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013def74-2b21-49ef-be87-f035f8ba5487-istiod-ca-cert\") pod \"013def74-2b21-49ef-be87-f035f8ba5487\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") "
Sep 09 20:40:44.001511 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.001382    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-istio-data\") pod \"013def74-2b21-49ef-be87-f035f8ba5487\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") "
Sep 09 20:40:44.001511 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.001415    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-credential-socket\") pod \"013def74-2b21-49ef-be87-f035f8ba5487\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") "
Sep 09 20:40:44.001511 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.001449    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-workload-socket\") pod \"013def74-2b21-49ef-be87-f035f8ba5487\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") "
Sep 09 20:40:44.001760 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.001559    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzbtc\" (UniqueName: \"kubernetes.io/projected/013def74-2b21-49ef-be87-f035f8ba5487-kube-api-access-tzbtc\") pod \"013def74-2b21-49ef-be87-f035f8ba5487\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") "
Sep 09 20:40:44.001760 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.001627    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-workload-certs\") pod \"013def74-2b21-49ef-be87-f035f8ba5487\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") "
Sep 09 20:40:44.001760 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.001652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-istio-envoy\") pod \"013def74-2b21-49ef-be87-f035f8ba5487\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") "
Sep 09 20:40:44.001760 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.001694    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013def74-2b21-49ef-be87-f035f8ba5487-istio-token\") pod \"013def74-2b21-49ef-be87-f035f8ba5487\" (UID: \"013def74-2b21-49ef-be87-f035f8ba5487\") "
Sep 09 20:40:44.001962 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.001928    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "013def74-2b21-49ef-be87-f035f8ba5487" (UID: "013def74-2b21-49ef-be87-f035f8ba5487"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:44.001962 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.001943    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "013def74-2b21-49ef-be87-f035f8ba5487" (UID: "013def74-2b21-49ef-be87-f035f8ba5487"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:44.002075 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.001983    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013def74-2b21-49ef-be87-f035f8ba5487-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "013def74-2b21-49ef-be87-f035f8ba5487" (UID: "013def74-2b21-49ef-be87-f035f8ba5487"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:44.002127 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.002097    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013def74-2b21-49ef-be87-f035f8ba5487-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:44.002127 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.002115    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:44.002236 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.002130    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:44.002457 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.002405    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-istio-data" (OuterVolumeSpecName: "istio-data") pod "013def74-2b21-49ef-be87-f035f8ba5487" (UID: "013def74-2b21-49ef-be87-f035f8ba5487"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:44.002607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.002469    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "013def74-2b21-49ef-be87-f035f8ba5487" (UID: "013def74-2b21-49ef-be87-f035f8ba5487"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:44.003637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.003613    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/013def74-2b21-49ef-be87-f035f8ba5487-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "013def74-2b21-49ef-be87-f035f8ba5487" (UID: "013def74-2b21-49ef-be87-f035f8ba5487"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:40:44.003880 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.003862    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013def74-2b21-49ef-be87-f035f8ba5487-istio-token" (OuterVolumeSpecName: "istio-token") pod "013def74-2b21-49ef-be87-f035f8ba5487" (UID: "013def74-2b21-49ef-be87-f035f8ba5487"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:44.004087 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.004064    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "013def74-2b21-49ef-be87-f035f8ba5487" (UID: "013def74-2b21-49ef-be87-f035f8ba5487"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:44.004199 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.004182    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013def74-2b21-49ef-be87-f035f8ba5487-kube-api-access-tzbtc" (OuterVolumeSpecName: "kube-api-access-tzbtc") pod "013def74-2b21-49ef-be87-f035f8ba5487" (UID: "013def74-2b21-49ef-be87-f035f8ba5487"). InnerVolumeSpecName "kube-api-access-tzbtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:44.103070 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.103028    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013def74-2b21-49ef-be87-f035f8ba5487-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:44.103070 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.103062    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:44.103070 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.103075    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzbtc\" (UniqueName: \"kubernetes.io/projected/013def74-2b21-49ef-be87-f035f8ba5487-kube-api-access-tzbtc\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:44.103529 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.103089    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:44.103529 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.103103    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013def74-2b21-49ef-be87-f035f8ba5487-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:44.103529 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.103112    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013def74-2b21-49ef-be87-f035f8ba5487-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:44.279207 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.279173    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"]
Sep 09 20:40:44.283991 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:44.283963    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-077e89-openshift-default-6cfc94b6bd-2p4gs"]
Sep 09 20:40:45.313372 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:45.313330    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="013def74-2b21-49ef-be87-f035f8ba5487" path="/var/lib/kubelet/pods/013def74-2b21-49ef-be87-f035f8ba5487/volumes"
Sep 09 20:40:51.738098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:51.738064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:53.298521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:53.298479    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:40:54.524865 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.524830    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"]
Sep 09 20:40:54.525370 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.525086    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl" podUID="276b4f79-c28f-4c68-9e20-10cf4e00eca9" containerName="maas-api" containerID="cri-o://44c68a1f21bdd9bcef09cab50fe0669186a5e5bd04fd20c3c279b225bad421f6" gracePeriod=30
Sep 09 20:40:54.554552 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.554501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:54.635135 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.635097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:54.714727 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.714687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:54.788868 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.788778    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:40:54.846613 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.846562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:54.904483 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.904446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-metrics-certs\") pod \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") "
Sep 09 20:40:54.904702 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.904497    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tm7f\" (UniqueName: \"kubernetes.io/projected/276b4f79-c28f-4c68-9e20-10cf4e00eca9-kube-api-access-7tm7f\") pod \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") "
Sep 09 20:40:54.904702 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.904587    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276b4f79-c28f-4c68-9e20-10cf4e00eca9-openshift-service-ca\") pod \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") "
Sep 09 20:40:54.904702 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.904619    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-maas-api-tls\") pod \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\" (UID: \"276b4f79-c28f-4c68-9e20-10cf4e00eca9\") "
Sep 09 20:40:54.904994 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.904963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276b4f79-c28f-4c68-9e20-10cf4e00eca9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "276b4f79-c28f-4c68-9e20-10cf4e00eca9" (UID: "276b4f79-c28f-4c68-9e20-10cf4e00eca9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:54.906848 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.906818    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "276b4f79-c28f-4c68-9e20-10cf4e00eca9" (UID: "276b4f79-c28f-4c68-9e20-10cf4e00eca9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:40:54.906848 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.906835    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276b4f79-c28f-4c68-9e20-10cf4e00eca9-kube-api-access-7tm7f" (OuterVolumeSpecName: "kube-api-access-7tm7f") pod "276b4f79-c28f-4c68-9e20-10cf4e00eca9" (UID: "276b4f79-c28f-4c68-9e20-10cf4e00eca9"). InnerVolumeSpecName "kube-api-access-7tm7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:54.907014 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.906849    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "276b4f79-c28f-4c68-9e20-10cf4e00eca9" (UID: "276b4f79-c28f-4c68-9e20-10cf4e00eca9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:40:54.921784 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:54.921738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:55.006059 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.006018    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:55.006059 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.006055    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tm7f\" (UniqueName: \"kubernetes.io/projected/276b4f79-c28f-4c68-9e20-10cf4e00eca9-kube-api-access-7tm7f\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:55.006270 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.006070    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276b4f79-c28f-4c68-9e20-10cf4e00eca9-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:55.006270 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.006082    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276b4f79-c28f-4c68-9e20-10cf4e00eca9-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:40:55.015922 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.015894    2563 generic.go:356] "Generic (PLEG): container finished" podID="276b4f79-c28f-4c68-9e20-10cf4e00eca9" containerID="44c68a1f21bdd9bcef09cab50fe0669186a5e5bd04fd20c3c279b225bad421f6" exitCode=0
Sep 09 20:40:55.016063 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.015943    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"
Sep 09 20:40:55.016063 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.015979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl" event={"ID":"276b4f79-c28f-4c68-9e20-10cf4e00eca9","Type":"ContainerDied","Data":"44c68a1f21bdd9bcef09cab50fe0669186a5e5bd04fd20c3c279b225bad421f6"}
Sep 09 20:40:55.016063 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.016028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl" event={"ID":"276b4f79-c28f-4c68-9e20-10cf4e00eca9","Type":"ContainerDied","Data":"f611c169ea10a491c2380c3cb32fdaf9ca1e74d738ca65ec6e7caee8187f90b0"}
Sep 09 20:40:55.016063 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.016055    2563 scope.go:117] "RemoveContainer" containerID="44c68a1f21bdd9bcef09cab50fe0669186a5e5bd04fd20c3c279b225bad421f6"
Sep 09 20:40:55.025949 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.025930    2563 scope.go:117] "RemoveContainer" containerID="44c68a1f21bdd9bcef09cab50fe0669186a5e5bd04fd20c3c279b225bad421f6"
Sep 09 20:40:55.026248 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:40:55.026229    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c68a1f21bdd9bcef09cab50fe0669186a5e5bd04fd20c3c279b225bad421f6\": container with ID starting with 44c68a1f21bdd9bcef09cab50fe0669186a5e5bd04fd20c3c279b225bad421f6 not found: ID does not exist" containerID="44c68a1f21bdd9bcef09cab50fe0669186a5e5bd04fd20c3c279b225bad421f6"
Sep 09 20:40:55.026311 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.026262    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c68a1f21bdd9bcef09cab50fe0669186a5e5bd04fd20c3c279b225bad421f6"} err="failed to get container status \"44c68a1f21bdd9bcef09cab50fe0669186a5e5bd04fd20c3c279b225bad421f6\": rpc error: code = NotFound desc = could not find container \"44c68a1f21bdd9bcef09cab50fe0669186a5e5bd04fd20c3c279b225bad421f6\": container with ID starting with 44c68a1f21bdd9bcef09cab50fe0669186a5e5bd04fd20c3c279b225bad421f6 not found: ID does not exist"
Sep 09 20:40:55.037830 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.037800    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"]
Sep 09 20:40:55.039593 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.039515    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-06f7dc-788d56695f-mttgl"]
Sep 09 20:40:55.174713 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.174683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:55.319937 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.319851    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276b4f79-c28f-4c68-9e20-10cf4e00eca9" path="/var/lib/kubelet/pods/276b4f79-c28f-4c68-9e20-10cf4e00eca9/volumes"
Sep 09 20:40:55.848118 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:55.848073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:56.570146 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:56.570110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:57.373668 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:57.373628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:58.171030 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:40:58.170995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:02.362249 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:02.362213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:02.460780 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:02.460745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:02.574682 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:02.574633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:03.240226 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:03.240184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:03.943776 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:03.943739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:04.665619 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.665587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:04.931413 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.931331    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7580-e5de37"]
Sep 09 20:41:04.931698 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.931685    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b72b963-3eb4-47df-9003-bb022a3a1fbd" containerName="maas-api"
Sep 09 20:41:04.931748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.931700    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b72b963-3eb4-47df-9003-bb022a3a1fbd" containerName="maas-api"
Sep 09 20:41:04.931748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.931710    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="013def74-2b21-49ef-be87-f035f8ba5487" containerName="istio-proxy"
Sep 09 20:41:04.931748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.931716    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="013def74-2b21-49ef-be87-f035f8ba5487" containerName="istio-proxy"
Sep 09 20:41:04.931748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.931725    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="013def74-2b21-49ef-be87-f035f8ba5487" containerName="istio-proxy"
Sep 09 20:41:04.931748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.931731    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="013def74-2b21-49ef-be87-f035f8ba5487" containerName="istio-proxy"
Sep 09 20:41:04.931748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.931738    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276b4f79-c28f-4c68-9e20-10cf4e00eca9" containerName="maas-api"
Sep 09 20:41:04.931748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.931743    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="276b4f79-c28f-4c68-9e20-10cf4e00eca9" containerName="maas-api"
Sep 09 20:41:04.931960 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.931805    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="013def74-2b21-49ef-be87-f035f8ba5487" containerName="istio-proxy"
Sep 09 20:41:04.931960 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.931813    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b72b963-3eb4-47df-9003-bb022a3a1fbd" containerName="maas-api"
Sep 09 20:41:04.931960 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.931820    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="276b4f79-c28f-4c68-9e20-10cf4e00eca9" containerName="maas-api"
Sep 09 20:41:04.934647 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.934630    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7580-e5de37"
Sep 09 20:41:04.937130 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.937099    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5fctx\""
Sep 09 20:41:04.939862 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:04.939837    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7580-e5de37"]
Sep 09 20:41:05.094182 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:05.094149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzqx2\" (UniqueName: \"kubernetes.io/projected/d413152b-eae2-4105-a9e0-e58fba9fa4da-kube-api-access-mzqx2\") pod \"test-curl-7580-e5de37\" (UID: \"d413152b-eae2-4105-a9e0-e58fba9fa4da\") " pod="opendatahub/test-curl-7580-e5de37"
Sep 09 20:41:05.195338 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:05.195231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzqx2\" (UniqueName: \"kubernetes.io/projected/d413152b-eae2-4105-a9e0-e58fba9fa4da-kube-api-access-mzqx2\") pod \"test-curl-7580-e5de37\" (UID: \"d413152b-eae2-4105-a9e0-e58fba9fa4da\") " pod="opendatahub/test-curl-7580-e5de37"
Sep 09 20:41:05.203664 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:05.203639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzqx2\" (UniqueName: \"kubernetes.io/projected/d413152b-eae2-4105-a9e0-e58fba9fa4da-kube-api-access-mzqx2\") pod \"test-curl-7580-e5de37\" (UID: \"d413152b-eae2-4105-a9e0-e58fba9fa4da\") " pod="opendatahub/test-curl-7580-e5de37"
Sep 09 20:41:05.245696 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:05.245662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7580-e5de37"
Sep 09 20:41:05.476369 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:05.476333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:05.513702 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:05.513678    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7580-e5de37"]
Sep 09 20:41:05.516408 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:41:05.516377    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd413152b_eae2_4105_a9e0_e58fba9fa4da.slice/crio-11f54423a04fff991405411858aa00e007c4e9074daf99c758693ec853cdec42 WatchSource:0}: Error finding container 11f54423a04fff991405411858aa00e007c4e9074daf99c758693ec853cdec42: Status 404 returned error can't find the container with id 11f54423a04fff991405411858aa00e007c4e9074daf99c758693ec853cdec42
Sep 09 20:41:06.070460 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:06.070427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7580-e5de37" event={"ID":"d413152b-eae2-4105-a9e0-e58fba9fa4da","Type":"ContainerStarted","Data":"902a947f2f5ff4f90bd77c907cda21b3cbc54b6bce020dee10c097d40d97ae7f"}
Sep 09 20:41:06.070677 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:06.070468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7580-e5de37" event={"ID":"d413152b-eae2-4105-a9e0-e58fba9fa4da","Type":"ContainerStarted","Data":"11f54423a04fff991405411858aa00e007c4e9074daf99c758693ec853cdec42"}
Sep 09 20:41:06.085721 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:06.085672    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7580-e5de37" podStartSLOduration=2.085655869 podStartE2EDuration="2.085655869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:06.084501641 +0000 UTC m=+1473.323603942" watchObservedRunningTime="2026-09-09 20:41:06.085655869 +0000 UTC m=+1473.324758168"
Sep 09 20:41:06.647713 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:06.647678    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7580-e5de37"]
Sep 09 20:41:06.654354 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:06.654327    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7580-e5de37"]
Sep 09 20:41:06.766680 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:06.766644    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7580-37a5c8"]
Sep 09 20:41:06.766993 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:06.766981    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d413152b-eae2-4105-a9e0-e58fba9fa4da" containerName="test-curl-7580-e5de37"
Sep 09 20:41:06.767042 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:06.766994    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d413152b-eae2-4105-a9e0-e58fba9fa4da" containerName="test-curl-7580-e5de37"
Sep 09 20:41:06.767078 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:06.767068    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d413152b-eae2-4105-a9e0-e58fba9fa4da" containerName="test-curl-7580-e5de37"
Sep 09 20:41:06.767078 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:06.767077    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="013def74-2b21-49ef-be87-f035f8ba5487" containerName="istio-proxy"
Sep 09 20:41:06.769981 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:06.769964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7580-37a5c8"
Sep 09 20:41:06.777459 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:06.777436    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7580-37a5c8"]
Sep 09 20:41:06.911456 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:06.911417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx6t7\" (UniqueName: \"kubernetes.io/projected/ca8b3667-4e1b-453e-8a79-2356e5086b76-kube-api-access-zx6t7\") pod \"test-curl-7580-37a5c8\" (UID: \"ca8b3667-4e1b-453e-8a79-2356e5086b76\") " pod="opendatahub/test-curl-7580-37a5c8"
Sep 09 20:41:07.011891 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:07.011857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx6t7\" (UniqueName: \"kubernetes.io/projected/ca8b3667-4e1b-453e-8a79-2356e5086b76-kube-api-access-zx6t7\") pod \"test-curl-7580-37a5c8\" (UID: \"ca8b3667-4e1b-453e-8a79-2356e5086b76\") " pod="opendatahub/test-curl-7580-37a5c8"
Sep 09 20:41:07.019876 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:07.019808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx6t7\" (UniqueName: \"kubernetes.io/projected/ca8b3667-4e1b-453e-8a79-2356e5086b76-kube-api-access-zx6t7\") pod \"test-curl-7580-37a5c8\" (UID: \"ca8b3667-4e1b-453e-8a79-2356e5086b76\") " pod="opendatahub/test-curl-7580-37a5c8"
Sep 09 20:41:07.079334 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:07.079192    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7580-37a5c8"
Sep 09 20:41:07.081833 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:41:07.081808    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7580-e5de37" container="test-curl-7580-e5de37"
Sep 09 20:41:07.347143 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:07.346988    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7580-37a5c8"]
Sep 09 20:41:07.349897 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:41:07.349855    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca8b3667_4e1b_453e_8a79_2356e5086b76.slice/crio-890a7f136ae0c792aa9d8db71e707e84dd118c95149d19ceee4ee2ce4c2ab799 WatchSource:0}: Error finding container 890a7f136ae0c792aa9d8db71e707e84dd118c95149d19ceee4ee2ce4c2ab799: Status 404 returned error can't find the container with id 890a7f136ae0c792aa9d8db71e707e84dd118c95149d19ceee4ee2ce4c2ab799
Sep 09 20:41:08.087482 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:08.087439    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7580-37a5c8" event={"ID":"ca8b3667-4e1b-453e-8a79-2356e5086b76","Type":"ContainerStarted","Data":"c49564f3eab715627e0b7068e43f7c476e0ac2b5918ffef534cf4c14bc5b1639"}
Sep 09 20:41:08.087946 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:08.087491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7580-37a5c8" event={"ID":"ca8b3667-4e1b-453e-8a79-2356e5086b76","Type":"ContainerStarted","Data":"890a7f136ae0c792aa9d8db71e707e84dd118c95149d19ceee4ee2ce4c2ab799"}
Sep 09 20:41:08.087946 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:08.087496    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7580-e5de37" podUID="d413152b-eae2-4105-a9e0-e58fba9fa4da" containerName="test-curl-7580-e5de37" containerID="cri-o://902a947f2f5ff4f90bd77c907cda21b3cbc54b6bce020dee10c097d40d97ae7f" gracePeriod=2
Sep 09 20:41:08.101635 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:08.101589    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7580-37a5c8" podStartSLOduration=2.101552723 podStartE2EDuration="2.101552723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:08.100644561 +0000 UTC m=+1475.339746861" watchObservedRunningTime="2026-09-09 20:41:08.101552723 +0000 UTC m=+1475.340655023"
Sep 09 20:41:08.682345 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:08.682310    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7580-37a5c8"]
Sep 09 20:41:08.691520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:08.691484    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7580-37a5c8"]
Sep 09 20:41:08.705314 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:08.705278    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:41:09.093411 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:41:09.093378    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7580-37a5c8" container="test-curl-7580-37a5c8"
Sep 09 20:41:10.097908 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:10.097874    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7580-37a5c8" podUID="ca8b3667-4e1b-453e-8a79-2356e5086b76" containerName="test-curl-7580-37a5c8" containerID="cri-o://c49564f3eab715627e0b7068e43f7c476e0ac2b5918ffef534cf4c14bc5b1639" gracePeriod=2
Sep 09 20:41:10.277175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:10.277148    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7580-e5de37"
Sep 09 20:41:10.442264 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:10.442180    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzqx2\" (UniqueName: \"kubernetes.io/projected/d413152b-eae2-4105-a9e0-e58fba9fa4da-kube-api-access-mzqx2\") pod \"d413152b-eae2-4105-a9e0-e58fba9fa4da\" (UID: \"d413152b-eae2-4105-a9e0-e58fba9fa4da\") "
Sep 09 20:41:10.444234 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:10.444202    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d413152b-eae2-4105-a9e0-e58fba9fa4da-kube-api-access-mzqx2" (OuterVolumeSpecName: "kube-api-access-mzqx2") pod "d413152b-eae2-4105-a9e0-e58fba9fa4da" (UID: "d413152b-eae2-4105-a9e0-e58fba9fa4da"). InnerVolumeSpecName "kube-api-access-mzqx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:10.543451 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:10.543409    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzqx2\" (UniqueName: \"kubernetes.io/projected/d413152b-eae2-4105-a9e0-e58fba9fa4da-kube-api-access-mzqx2\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:10.745375 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:10.745336    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"]
Sep 09 20:41:10.766255 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:10.766225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:10.866992 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:10.866948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:11.104188 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:11.104089    2563 generic.go:356] "Generic (PLEG): container finished" podID="d413152b-eae2-4105-a9e0-e58fba9fa4da" containerID="902a947f2f5ff4f90bd77c907cda21b3cbc54b6bce020dee10c097d40d97ae7f" exitCode=137
Sep 09 20:41:11.104188 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:11.104140    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7580-e5de37"
Sep 09 20:41:11.104188 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:11.104166    2563 scope.go:117] "RemoveContainer" containerID="902a947f2f5ff4f90bd77c907cda21b3cbc54b6bce020dee10c097d40d97ae7f"
Sep 09 20:41:11.112991 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:11.112970    2563 scope.go:117] "RemoveContainer" containerID="902a947f2f5ff4f90bd77c907cda21b3cbc54b6bce020dee10c097d40d97ae7f"
Sep 09 20:41:11.113276 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:41:11.113257    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902a947f2f5ff4f90bd77c907cda21b3cbc54b6bce020dee10c097d40d97ae7f\": container with ID starting with 902a947f2f5ff4f90bd77c907cda21b3cbc54b6bce020dee10c097d40d97ae7f not found: ID does not exist" containerID="902a947f2f5ff4f90bd77c907cda21b3cbc54b6bce020dee10c097d40d97ae7f"
Sep 09 20:41:11.113333 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:11.113286    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902a947f2f5ff4f90bd77c907cda21b3cbc54b6bce020dee10c097d40d97ae7f"} err="failed to get container status \"902a947f2f5ff4f90bd77c907cda21b3cbc54b6bce020dee10c097d40d97ae7f\": rpc error: code = NotFound desc = could not find container \"902a947f2f5ff4f90bd77c907cda21b3cbc54b6bce020dee10c097d40d97ae7f\": container with ID starting with 902a947f2f5ff4f90bd77c907cda21b3cbc54b6bce020dee10c097d40d97ae7f not found: ID does not exist"
Sep 09 20:41:11.114968 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:41:11.114944    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7580-e5de37\" is forbidden: User \"system:node:ip-10-0-143-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-241.ec2.internal' and this object" podUID="d413152b-eae2-4105-a9e0-e58fba9fa4da" pod="opendatahub/test-curl-7580-e5de37"
Sep 09 20:41:11.313375 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:11.313340    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d413152b-eae2-4105-a9e0-e58fba9fa4da" path="/var/lib/kubelet/pods/d413152b-eae2-4105-a9e0-e58fba9fa4da/volumes"
Sep 09 20:41:11.673411 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:11.673363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:11.988221 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:11.988179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:12.288734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.288708    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7580-37a5c8"
Sep 09 20:41:12.459819 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.459783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx6t7\" (UniqueName: \"kubernetes.io/projected/ca8b3667-4e1b-453e-8a79-2356e5086b76-kube-api-access-zx6t7\") pod \"ca8b3667-4e1b-453e-8a79-2356e5086b76\" (UID: \"ca8b3667-4e1b-453e-8a79-2356e5086b76\") "
Sep 09 20:41:12.462114 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.462078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8b3667-4e1b-453e-8a79-2356e5086b76-kube-api-access-zx6t7" (OuterVolumeSpecName: "kube-api-access-zx6t7") pod "ca8b3667-4e1b-453e-8a79-2356e5086b76" (UID: "ca8b3667-4e1b-453e-8a79-2356e5086b76"). InnerVolumeSpecName "kube-api-access-zx6t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:12.560888 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.560804    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx6t7\" (UniqueName: \"kubernetes.io/projected/ca8b3667-4e1b-453e-8a79-2356e5086b76-kube-api-access-zx6t7\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:12.651151 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.651112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:12.680874 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.680838    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"]
Sep 09 20:41:12.681141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.681112    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft" podUID="c9849330-7ffc-4835-bfd6-feec46672d6c" containerName="payload-processing" containerID="cri-o://cbc1a244c63c12a9486bce27a1f9a71436d1e4caa2a711f63398304256248c48" gracePeriod=30
Sep 09 20:41:12.688293 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.688261    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"]
Sep 09 20:41:12.688525 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.688493    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw" podUID="89a749bb-51b5-4a69-b3ef-9fba029fce33" containerName="payload-pre-processing" containerID="cri-o://01a633b54401552e2cbb8fd889958c30e43143645d937babe4b754758e678ace" gracePeriod=30
Sep 09 20:41:12.752012 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.751962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:12.844264 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.844094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:12.904006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.903972    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:12.926286 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.926258    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"
Sep 09 20:41:12.967062 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:12.967030    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"
Sep 09 20:41:13.067123 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.067082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a749bb-51b5-4a69-b3ef-9fba029fce33-plugins-config-volume\") pod \"89a749bb-51b5-4a69-b3ef-9fba029fce33\" (UID: \"89a749bb-51b5-4a69-b3ef-9fba029fce33\") "
Sep 09 20:41:13.067304 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.067210    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwngs\" (UniqueName: \"kubernetes.io/projected/89a749bb-51b5-4a69-b3ef-9fba029fce33-kube-api-access-vwngs\") pod \"89a749bb-51b5-4a69-b3ef-9fba029fce33\" (UID: \"89a749bb-51b5-4a69-b3ef-9fba029fce33\") "
Sep 09 20:41:13.067304 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.067249    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw77q\" (UniqueName: \"kubernetes.io/projected/c9849330-7ffc-4835-bfd6-feec46672d6c-kube-api-access-nw77q\") pod \"c9849330-7ffc-4835-bfd6-feec46672d6c\" (UID: \"c9849330-7ffc-4835-bfd6-feec46672d6c\") "
Sep 09 20:41:13.067304 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.067293    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9849330-7ffc-4835-bfd6-feec46672d6c-plugins-config-volume\") pod \"c9849330-7ffc-4835-bfd6-feec46672d6c\" (UID: \"c9849330-7ffc-4835-bfd6-feec46672d6c\") "
Sep 09 20:41:13.067525 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.067502    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a749bb-51b5-4a69-b3ef-9fba029fce33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89a749bb-51b5-4a69-b3ef-9fba029fce33" (UID: "89a749bb-51b5-4a69-b3ef-9fba029fce33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:13.067754 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.067724    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9849330-7ffc-4835-bfd6-feec46672d6c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9849330-7ffc-4835-bfd6-feec46672d6c" (UID: "c9849330-7ffc-4835-bfd6-feec46672d6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:13.069374 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.069353    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9849330-7ffc-4835-bfd6-feec46672d6c-kube-api-access-nw77q" (OuterVolumeSpecName: "kube-api-access-nw77q") pod "c9849330-7ffc-4835-bfd6-feec46672d6c" (UID: "c9849330-7ffc-4835-bfd6-feec46672d6c"). InnerVolumeSpecName "kube-api-access-nw77q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:13.069482 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.069432    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a749bb-51b5-4a69-b3ef-9fba029fce33-kube-api-access-vwngs" (OuterVolumeSpecName: "kube-api-access-vwngs") pod "89a749bb-51b5-4a69-b3ef-9fba029fce33" (UID: "89a749bb-51b5-4a69-b3ef-9fba029fce33"). InnerVolumeSpecName "kube-api-access-vwngs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:13.117324 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.117289    2563 generic.go:356] "Generic (PLEG): container finished" podID="89a749bb-51b5-4a69-b3ef-9fba029fce33" containerID="01a633b54401552e2cbb8fd889958c30e43143645d937babe4b754758e678ace" exitCode=0
Sep 09 20:41:13.117463 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.117349    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"
Sep 09 20:41:13.117463 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.117390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw" event={"ID":"89a749bb-51b5-4a69-b3ef-9fba029fce33","Type":"ContainerDied","Data":"01a633b54401552e2cbb8fd889958c30e43143645d937babe4b754758e678ace"}
Sep 09 20:41:13.117463 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.117444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw" event={"ID":"89a749bb-51b5-4a69-b3ef-9fba029fce33","Type":"ContainerDied","Data":"bf739125a3bab876e42272a34b9b4c94eb2ca2640c04a18bf1b2cdd8f9035067"}
Sep 09 20:41:13.117679 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.117472    2563 scope.go:117] "RemoveContainer" containerID="01a633b54401552e2cbb8fd889958c30e43143645d937babe4b754758e678ace"
Sep 09 20:41:13.118883 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.118860    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9849330-7ffc-4835-bfd6-feec46672d6c" containerID="cbc1a244c63c12a9486bce27a1f9a71436d1e4caa2a711f63398304256248c48" exitCode=0
Sep 09 20:41:13.118978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.118901    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"
Sep 09 20:41:13.118978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.118934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft" event={"ID":"c9849330-7ffc-4835-bfd6-feec46672d6c","Type":"ContainerDied","Data":"cbc1a244c63c12a9486bce27a1f9a71436d1e4caa2a711f63398304256248c48"}
Sep 09 20:41:13.118978 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.118971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft" event={"ID":"c9849330-7ffc-4835-bfd6-feec46672d6c","Type":"ContainerDied","Data":"2803eb05b67d4ada89b5f0b7425fe6129f65d8b33589994114b4ff852535464b"}
Sep 09 20:41:13.120283 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.120226    2563 generic.go:356] "Generic (PLEG): container finished" podID="ca8b3667-4e1b-453e-8a79-2356e5086b76" containerID="c49564f3eab715627e0b7068e43f7c476e0ac2b5918ffef534cf4c14bc5b1639" exitCode=137
Sep 09 20:41:13.120283 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.120263    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7580-37a5c8"
Sep 09 20:41:13.127079 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.127057    2563 scope.go:117] "RemoveContainer" containerID="01a633b54401552e2cbb8fd889958c30e43143645d937babe4b754758e678ace"
Sep 09 20:41:13.127388 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:41:13.127365    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a633b54401552e2cbb8fd889958c30e43143645d937babe4b754758e678ace\": container with ID starting with 01a633b54401552e2cbb8fd889958c30e43143645d937babe4b754758e678ace not found: ID does not exist" containerID="01a633b54401552e2cbb8fd889958c30e43143645d937babe4b754758e678ace"
Sep 09 20:41:13.127460 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.127398    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a633b54401552e2cbb8fd889958c30e43143645d937babe4b754758e678ace"} err="failed to get container status \"01a633b54401552e2cbb8fd889958c30e43143645d937babe4b754758e678ace\": rpc error: code = NotFound desc = could not find container \"01a633b54401552e2cbb8fd889958c30e43143645d937babe4b754758e678ace\": container with ID starting with 01a633b54401552e2cbb8fd889958c30e43143645d937babe4b754758e678ace not found: ID does not exist"
Sep 09 20:41:13.127460 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.127416    2563 scope.go:117] "RemoveContainer" containerID="cbc1a244c63c12a9486bce27a1f9a71436d1e4caa2a711f63398304256248c48"
Sep 09 20:41:13.136991 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.136972    2563 scope.go:117] "RemoveContainer" containerID="cbc1a244c63c12a9486bce27a1f9a71436d1e4caa2a711f63398304256248c48"
Sep 09 20:41:13.137282 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:41:13.137261    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc1a244c63c12a9486bce27a1f9a71436d1e4caa2a711f63398304256248c48\": container with ID starting with cbc1a244c63c12a9486bce27a1f9a71436d1e4caa2a711f63398304256248c48 not found: ID does not exist" containerID="cbc1a244c63c12a9486bce27a1f9a71436d1e4caa2a711f63398304256248c48"
Sep 09 20:41:13.137355 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.137293    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc1a244c63c12a9486bce27a1f9a71436d1e4caa2a711f63398304256248c48"} err="failed to get container status \"cbc1a244c63c12a9486bce27a1f9a71436d1e4caa2a711f63398304256248c48\": rpc error: code = NotFound desc = could not find container \"cbc1a244c63c12a9486bce27a1f9a71436d1e4caa2a711f63398304256248c48\": container with ID starting with cbc1a244c63c12a9486bce27a1f9a71436d1e4caa2a711f63398304256248c48 not found: ID does not exist"
Sep 09 20:41:13.137355 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.137345    2563 scope.go:117] "RemoveContainer" containerID="c49564f3eab715627e0b7068e43f7c476e0ac2b5918ffef534cf4c14bc5b1639"
Sep 09 20:41:13.144984 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.144967    2563 scope.go:117] "RemoveContainer" containerID="c49564f3eab715627e0b7068e43f7c476e0ac2b5918ffef534cf4c14bc5b1639"
Sep 09 20:41:13.145290 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:41:13.145271    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49564f3eab715627e0b7068e43f7c476e0ac2b5918ffef534cf4c14bc5b1639\": container with ID starting with c49564f3eab715627e0b7068e43f7c476e0ac2b5918ffef534cf4c14bc5b1639 not found: ID does not exist" containerID="c49564f3eab715627e0b7068e43f7c476e0ac2b5918ffef534cf4c14bc5b1639"
Sep 09 20:41:13.145358 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.145295    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49564f3eab715627e0b7068e43f7c476e0ac2b5918ffef534cf4c14bc5b1639"} err="failed to get container status \"c49564f3eab715627e0b7068e43f7c476e0ac2b5918ffef534cf4c14bc5b1639\": rpc error: code = NotFound desc = could not find container \"c49564f3eab715627e0b7068e43f7c476e0ac2b5918ffef534cf4c14bc5b1639\": container with ID starting with c49564f3eab715627e0b7068e43f7c476e0ac2b5918ffef534cf4c14bc5b1639 not found: ID does not exist"
Sep 09 20:41:13.146110 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.146090    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"]
Sep 09 20:41:13.149872 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.149848    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc2ebb-586f8b489-g2cft"]
Sep 09 20:41:13.165114 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.165084    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"]
Sep 09 20:41:13.168337 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.168314    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a749bb-51b5-4a69-b3ef-9fba029fce33-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:13.168337 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.168337    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwngs\" (UniqueName: \"kubernetes.io/projected/89a749bb-51b5-4a69-b3ef-9fba029fce33-kube-api-access-vwngs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:13.168510 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.168350    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw77q\" (UniqueName: \"kubernetes.io/projected/c9849330-7ffc-4835-bfd6-feec46672d6c-kube-api-access-nw77q\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:13.168510 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.168359    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9849330-7ffc-4835-bfd6-feec46672d6c-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:13.168510 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:41:13.168463    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7580-37a5c8\" is forbidden: User \"system:node:ip-10-0-143-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-241.ec2.internal' and this object" podUID="ca8b3667-4e1b-453e-8a79-2356e5086b76" pod="opendatahub/test-curl-7580-37a5c8"
Sep 09 20:41:13.168752 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.168733    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc2ebb-67b865dc4b-8mxrw"]
Sep 09 20:41:13.196664 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.196620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:13.298385 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.298347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:13.312086 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:41:13.312053    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7580-37a5c8\" is forbidden: User \"system:node:ip-10-0-143-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-241.ec2.internal' and this object" podUID="ca8b3667-4e1b-453e-8a79-2356e5086b76" pod="opendatahub/test-curl-7580-37a5c8"
Sep 09 20:41:13.314040 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.314017    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89a749bb-51b5-4a69-b3ef-9fba029fce33" path="/var/lib/kubelet/pods/89a749bb-51b5-4a69-b3ef-9fba029fce33/volumes"
Sep 09 20:41:13.314379 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.314366    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9849330-7ffc-4835-bfd6-feec46672d6c" path="/var/lib/kubelet/pods/c9849330-7ffc-4835-bfd6-feec46672d6c/volumes"
Sep 09 20:41:13.314665 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:13.314653    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca8b3667-4e1b-453e-8a79-2356e5086b76" path="/var/lib/kubelet/pods/ca8b3667-4e1b-453e-8a79-2356e5086b76/volumes"
Sep 09 20:41:14.067826 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:14.067767    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:14.870667 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:14.870627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:15.576826 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:15.576794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:19.156521 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:19.156483    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:19.277885 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:19.277851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:19.398668 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:19.398637    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:19.998355 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:19.998320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:21.002533 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:21.002499    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:21.793076 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:21.793046    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:22.107821 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:22.107730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:41:22.234405 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:22.234370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:22.571590 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:22.571539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:23.372402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:23.372369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:25.765773 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:25.765726    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk" podUID="cc38f83d-11f6-45c3-9c69-b0504ce6be58" containerName="main" containerID="cri-o://26847d4187e682a8cc5660e58df8d5df9e807d066bcd9b98f52db70820a78239" gracePeriod=45
Sep 09 20:41:26.010381 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.010352    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:41:26.079314 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.079219    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-tmp-dir\") pod \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") "
Sep 09 20:41:26.079314 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.079270    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc38f83d-11f6-45c3-9c69-b0504ce6be58-tls-certs\") pod \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") "
Sep 09 20:41:26.079314 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.079291    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-dshm\") pod \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") "
Sep 09 20:41:26.079314 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.079317    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-model-cache\") pod \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") "
Sep 09 20:41:26.079710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.079355    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-home\") pod \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") "
Sep 09 20:41:26.079710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.079393    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc24z\" (UniqueName: \"kubernetes.io/projected/cc38f83d-11f6-45c3-9c69-b0504ce6be58-kube-api-access-tc24z\") pod \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\" (UID: \"cc38f83d-11f6-45c3-9c69-b0504ce6be58\") "
Sep 09 20:41:26.079710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.079515    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc38f83d-11f6-45c3-9c69-b0504ce6be58" (UID: "cc38f83d-11f6-45c3-9c69-b0504ce6be58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:26.079710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.079635    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-tmp-dir\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:26.079710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.079658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc38f83d-11f6-45c3-9c69-b0504ce6be58" (UID: "cc38f83d-11f6-45c3-9c69-b0504ce6be58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:26.079710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.079669    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-home" (OuterVolumeSpecName: "home") pod "cc38f83d-11f6-45c3-9c69-b0504ce6be58" (UID: "cc38f83d-11f6-45c3-9c69-b0504ce6be58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:26.081520 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.081487    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc38f83d-11f6-45c3-9c69-b0504ce6be58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc38f83d-11f6-45c3-9c69-b0504ce6be58" (UID: "cc38f83d-11f6-45c3-9c69-b0504ce6be58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:41:26.081636 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.081536    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-dshm" (OuterVolumeSpecName: "dshm") pod "cc38f83d-11f6-45c3-9c69-b0504ce6be58" (UID: "cc38f83d-11f6-45c3-9c69-b0504ce6be58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:26.081636 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.081589    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc38f83d-11f6-45c3-9c69-b0504ce6be58-kube-api-access-tc24z" (OuterVolumeSpecName: "kube-api-access-tc24z") pod "cc38f83d-11f6-45c3-9c69-b0504ce6be58" (UID: "cc38f83d-11f6-45c3-9c69-b0504ce6be58"). InnerVolumeSpecName "kube-api-access-tc24z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:26.180138 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.180098    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-home\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:26.180138 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.180134    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc24z\" (UniqueName: \"kubernetes.io/projected/cc38f83d-11f6-45c3-9c69-b0504ce6be58-kube-api-access-tc24z\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:26.180343 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.180149    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc38f83d-11f6-45c3-9c69-b0504ce6be58-tls-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:26.180343 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.180162    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-dshm\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:26.180343 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.180172    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc38f83d-11f6-45c3-9c69-b0504ce6be58-model-cache\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:26.192256 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.192221    2563 generic.go:356] "Generic (PLEG): container finished" podID="cc38f83d-11f6-45c3-9c69-b0504ce6be58" containerID="26847d4187e682a8cc5660e58df8d5df9e807d066bcd9b98f52db70820a78239" exitCode=0
Sep 09 20:41:26.192387 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.192267    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"
Sep 09 20:41:26.192387 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.192301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk" event={"ID":"cc38f83d-11f6-45c3-9c69-b0504ce6be58","Type":"ContainerDied","Data":"26847d4187e682a8cc5660e58df8d5df9e807d066bcd9b98f52db70820a78239"}
Sep 09 20:41:26.192387 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.192340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk" event={"ID":"cc38f83d-11f6-45c3-9c69-b0504ce6be58","Type":"ContainerDied","Data":"72ca09074b4fc0f0c3c7eefaef1a1f9b51bc7139eb8e9457f86cbcb36e768584"}
Sep 09 20:41:26.192387 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.192361    2563 scope.go:117] "RemoveContainer" containerID="26847d4187e682a8cc5660e58df8d5df9e807d066bcd9b98f52db70820a78239"
Sep 09 20:41:26.201191 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.201172    2563 scope.go:117] "RemoveContainer" containerID="26847d4187e682a8cc5660e58df8d5df9e807d066bcd9b98f52db70820a78239"
Sep 09 20:41:26.201451 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:41:26.201434    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26847d4187e682a8cc5660e58df8d5df9e807d066bcd9b98f52db70820a78239\": container with ID starting with 26847d4187e682a8cc5660e58df8d5df9e807d066bcd9b98f52db70820a78239 not found: ID does not exist" containerID="26847d4187e682a8cc5660e58df8d5df9e807d066bcd9b98f52db70820a78239"
Sep 09 20:41:26.201493 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.201460    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26847d4187e682a8cc5660e58df8d5df9e807d066bcd9b98f52db70820a78239"} err="failed to get container status \"26847d4187e682a8cc5660e58df8d5df9e807d066bcd9b98f52db70820a78239\": rpc error: code = NotFound desc = could not find container \"26847d4187e682a8cc5660e58df8d5df9e807d066bcd9b98f52db70820a78239\": container with ID starting with 26847d4187e682a8cc5660e58df8d5df9e807d066bcd9b98f52db70820a78239 not found: ID does not exist"
Sep 09 20:41:26.212781 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.212757    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"]
Sep 09 20:41:26.215903 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:26.215872    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/sub-test-model-166b94-kserve-6c75547595-d24nk"]
Sep 09 20:41:27.313985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:27.313953    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc38f83d-11f6-45c3-9c69-b0504ce6be58" path="/var/lib/kubelet/pods/cc38f83d-11f6-45c3-9c69-b0504ce6be58/volumes"
Sep 09 20:41:28.012903 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:28.012865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:29.974116 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:29.974075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:30.523211 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.523170    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"]
Sep 09 20:41:30.523488 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.523441    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs" podUID="dce4d80e-a5b2-456e-acd8-772c3bf9dd09" containerName="maas-api" containerID="cri-o://a6466a02222df3df12cf2b2253e1a680f8fb836fc2cf15598e877907e3af0f39" gracePeriod=30
Sep 09 20:41:30.552699 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.552665    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:30.627651 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.627610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:30.689148 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.689109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:30.762006 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.761977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:30.771659 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.771640    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:41:30.816354 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.816262    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ckpk\" (UniqueName: \"kubernetes.io/projected/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-kube-api-access-4ckpk\") pod \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") "
Sep 09 20:41:30.816354 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.816307    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-metrics-certs\") pod \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") "
Sep 09 20:41:30.816585 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.816388    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-openshift-service-ca\") pod \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") "
Sep 09 20:41:30.816585 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.816417    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-maas-api-tls\") pod \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\" (UID: \"dce4d80e-a5b2-456e-acd8-772c3bf9dd09\") "
Sep 09 20:41:30.816865 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.816834    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dce4d80e-a5b2-456e-acd8-772c3bf9dd09" (UID: "dce4d80e-a5b2-456e-acd8-772c3bf9dd09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:30.819017 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.818981    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-kube-api-access-4ckpk" (OuterVolumeSpecName: "kube-api-access-4ckpk") pod "dce4d80e-a5b2-456e-acd8-772c3bf9dd09" (UID: "dce4d80e-a5b2-456e-acd8-772c3bf9dd09"). InnerVolumeSpecName "kube-api-access-4ckpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:30.819134 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.819061    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dce4d80e-a5b2-456e-acd8-772c3bf9dd09" (UID: "dce4d80e-a5b2-456e-acd8-772c3bf9dd09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:41:30.819134 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.819078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dce4d80e-a5b2-456e-acd8-772c3bf9dd09" (UID: "dce4d80e-a5b2-456e-acd8-772c3bf9dd09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:41:30.836493 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.836448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:30.917679 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.917637    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ckpk\" (UniqueName: \"kubernetes.io/projected/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-kube-api-access-4ckpk\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:30.917679 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.917672    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:30.917679 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.917685    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:30.917986 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:30.917696    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce4d80e-a5b2-456e-acd8-772c3bf9dd09-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:41:31.136383 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:31.136288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:31.218100 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:31.218061    2563 generic.go:356] "Generic (PLEG): container finished" podID="dce4d80e-a5b2-456e-acd8-772c3bf9dd09" containerID="a6466a02222df3df12cf2b2253e1a680f8fb836fc2cf15598e877907e3af0f39" exitCode=0
Sep 09 20:41:31.218270 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:31.218106    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"
Sep 09 20:41:31.218270 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:31.218140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs" event={"ID":"dce4d80e-a5b2-456e-acd8-772c3bf9dd09","Type":"ContainerDied","Data":"a6466a02222df3df12cf2b2253e1a680f8fb836fc2cf15598e877907e3af0f39"}
Sep 09 20:41:31.218270 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:31.218188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs" event={"ID":"dce4d80e-a5b2-456e-acd8-772c3bf9dd09","Type":"ContainerDied","Data":"1f6643431befaf440af74d6b0ef1a3085d6fe4274dbafd829dee52bab6998b1e"}
Sep 09 20:41:31.218270 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:31.218212    2563 scope.go:117] "RemoveContainer" containerID="a6466a02222df3df12cf2b2253e1a680f8fb836fc2cf15598e877907e3af0f39"
Sep 09 20:41:31.227698 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:31.227680    2563 scope.go:117] "RemoveContainer" containerID="a6466a02222df3df12cf2b2253e1a680f8fb836fc2cf15598e877907e3af0f39"
Sep 09 20:41:31.227997 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:41:31.227975    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6466a02222df3df12cf2b2253e1a680f8fb836fc2cf15598e877907e3af0f39\": container with ID starting with a6466a02222df3df12cf2b2253e1a680f8fb836fc2cf15598e877907e3af0f39 not found: ID does not exist" containerID="a6466a02222df3df12cf2b2253e1a680f8fb836fc2cf15598e877907e3af0f39"
Sep 09 20:41:31.228045 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:31.228006    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6466a02222df3df12cf2b2253e1a680f8fb836fc2cf15598e877907e3af0f39"} err="failed to get container status \"a6466a02222df3df12cf2b2253e1a680f8fb836fc2cf15598e877907e3af0f39\": rpc error: code = NotFound desc = could not find container \"a6466a02222df3df12cf2b2253e1a680f8fb836fc2cf15598e877907e3af0f39\": container with ID starting with a6466a02222df3df12cf2b2253e1a680f8fb836fc2cf15598e877907e3af0f39 not found: ID does not exist"
Sep 09 20:41:31.244374 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:31.244342    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"]
Sep 09 20:41:31.247514 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:31.247491    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-47e11e-df4b7b54c-xbkhs"]
Sep 09 20:41:31.270704 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:31.270674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:31.313294 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:31.313254    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dce4d80e-a5b2-456e-acd8-772c3bf9dd09" path="/var/lib/kubelet/pods/dce4d80e-a5b2-456e-acd8-772c3bf9dd09/volumes"
Sep 09 20:41:32.144858 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:32.144823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:32.983658 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:32.983615    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:33.450834 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:33.450749    2563 scope.go:117] "RemoveContainer" containerID="200ec5154ec8ff4d47850ebd044289faee9f82cefd8a9b6ed238f0a95805a77a"
Sep 09 20:41:34.001772 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.001738    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:41:34.164566 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.164531    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"]
Sep 09 20:41:34.164889 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.164875    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc38f83d-11f6-45c3-9c69-b0504ce6be58" containerName="main"
Sep 09 20:41:34.164940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.164891    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc38f83d-11f6-45c3-9c69-b0504ce6be58" containerName="main"
Sep 09 20:41:34.164940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.164902    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce4d80e-a5b2-456e-acd8-772c3bf9dd09" containerName="maas-api"
Sep 09 20:41:34.164940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.164910    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce4d80e-a5b2-456e-acd8-772c3bf9dd09" containerName="maas-api"
Sep 09 20:41:34.164940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.164921    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca8b3667-4e1b-453e-8a79-2356e5086b76" containerName="test-curl-7580-37a5c8"
Sep 09 20:41:34.164940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.164929    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca8b3667-4e1b-453e-8a79-2356e5086b76" containerName="test-curl-7580-37a5c8"
Sep 09 20:41:34.165098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.164945    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9849330-7ffc-4835-bfd6-feec46672d6c" containerName="payload-processing"
Sep 09 20:41:34.165098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.164950    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9849330-7ffc-4835-bfd6-feec46672d6c" containerName="payload-processing"
Sep 09 20:41:34.165098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.164957    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a749bb-51b5-4a69-b3ef-9fba029fce33" containerName="payload-pre-processing"
Sep 09 20:41:34.165098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.164962    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a749bb-51b5-4a69-b3ef-9fba029fce33" containerName="payload-pre-processing"
Sep 09 20:41:34.165098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.165019    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc38f83d-11f6-45c3-9c69-b0504ce6be58" containerName="main"
Sep 09 20:41:34.165098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.165027    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca8b3667-4e1b-453e-8a79-2356e5086b76" containerName="test-curl-7580-37a5c8"
Sep 09 20:41:34.165098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.165035    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce4d80e-a5b2-456e-acd8-772c3bf9dd09" containerName="maas-api"
Sep 09 20:41:34.165098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.165041    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9849330-7ffc-4835-bfd6-feec46672d6c" containerName="payload-processing"
Sep 09 20:41:34.165098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.165048    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="89a749bb-51b5-4a69-b3ef-9fba029fce33" containerName="payload-pre-processing"
Sep 09 20:41:34.169686 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.169664    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.173316 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.173270    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-166b94\"/\"default-dockercfg-9zkh8\""
Sep 09 20:41:34.173474 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.173322    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-166b94\"/\"kube-root-ca.crt\""
Sep 09 20:41:34.173474 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.173379    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-166b94\"/\"openshift-service-ca.crt\""
Sep 09 20:41:34.173474 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.173270    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-166b94\"/\"rate-test-model-166b94-kserve-self-signed-certs\""
Sep 09 20:41:34.179197 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.179168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"]
Sep 09 20:41:34.197022 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.196993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:34.247846 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.247807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-model-cache\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.247846 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.247851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0147a28-33b1-4b27-a071-1bf4572db820-tls-certs\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.248120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.247888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-home\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.248120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.248007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-dshm\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.248120 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.248063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-tmp-dir\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.248251 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.248123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbkmp\" (UniqueName: \"kubernetes.io/projected/f0147a28-33b1-4b27-a071-1bf4572db820-kube-api-access-bbkmp\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.349119 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.349012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-home\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.349119 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.349077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-dshm\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.349119 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.349116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-tmp-dir\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.349421 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.349163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbkmp\" (UniqueName: \"kubernetes.io/projected/f0147a28-33b1-4b27-a071-1bf4572db820-kube-api-access-bbkmp\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.349421 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.349198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-model-cache\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.349421 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.349223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0147a28-33b1-4b27-a071-1bf4572db820-tls-certs\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.349607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.349475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-home\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.349607 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.349491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-tmp-dir\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.349757 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.349735    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-model-cache\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.352128 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.352100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-dshm\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.352315 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.352297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0147a28-33b1-4b27-a071-1bf4572db820-tls-certs\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.359200 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.359167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbkmp\" (UniqueName: \"kubernetes.io/projected/f0147a28-33b1-4b27-a071-1bf4572db820-kube-api-access-bbkmp\") pod \"rate-test-model-166b94-kserve-688768848c-6tk87\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") " pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.480305 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.480267    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:34.750853 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:34.750828    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"]
Sep 09 20:41:34.753975 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:41:34.753938    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0147a28_33b1_4b27_a071_1bf4572db820.slice/crio-f94d868b37c43ee74e764c9ffb08e78ff13a3da91807090daf15cb4bb39f87e7 WatchSource:0}: Error finding container f94d868b37c43ee74e764c9ffb08e78ff13a3da91807090daf15cb4bb39f87e7: Status 404 returned error can't find the container with id f94d868b37c43ee74e764c9ffb08e78ff13a3da91807090daf15cb4bb39f87e7
Sep 09 20:41:35.238410 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:35.238370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87" event={"ID":"f0147a28-33b1-4b27-a071-1bf4572db820","Type":"ContainerStarted","Data":"89e889d10e1e1e94517c4d9f5d5362019a2536cbb8a610f8f3db5f42e1541eab"}
Sep 09 20:41:35.238410 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:35.238415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87" event={"ID":"f0147a28-33b1-4b27-a071-1bf4572db820","Type":"ContainerStarted","Data":"f94d868b37c43ee74e764c9ffb08e78ff13a3da91807090daf15cb4bb39f87e7"}
Sep 09 20:41:35.257331 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:35.257277    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87" podStartSLOduration=1.257258648 podStartE2EDuration="1.257258648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:35.256676011 +0000 UTC m=+1502.495778312" watchObservedRunningTime="2026-09-09 20:41:35.257258648 +0000 UTC m=+1502.496360951"
Sep 09 20:41:37.051345 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:37.051292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:37.150684 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:37.150647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:37.298007 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:37.297976    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:44.017687 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:44.017649    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:41:44.480731 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:44.480687    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:44.480890 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:44.480754    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:44.493213 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:44.493187    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:45.306361 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:45.306328    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:41:50.119985 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:41:50.119948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:02.646109 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:02.646024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:06.739765 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:06.739720    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:42:15.746404 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:15.746363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:16.883167 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:16.883132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:19.902308 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:19.902270    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:42:29.845658 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:29.845622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:30.506836 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:30.506805    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:42:37.420431 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:37.420386    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"]
Sep 09 20:42:37.440062 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:37.440034    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:38.279484 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:38.279445    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:38.706273 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:38.706237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:39.009528 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.009435    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"]
Sep 09 20:42:39.012886 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.012856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.015558 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.015538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-868e40-openshift-default-dockercfg-h2n6n\""
Sep 09 20:42:39.025333 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.025309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"]
Sep 09 20:42:39.037092 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.037067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:39.112773 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.112732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-workload-certs\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.112975 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.112786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnfr4\" (UniqueName: \"kubernetes.io/projected/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-kube-api-access-cnfr4\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.112975 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.112820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-credential-socket\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.112975 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.112845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-data\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.113169 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.113002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-podinfo\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.113169 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.113054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istiod-ca-cert\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.113169 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.113102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-token\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.113169 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.113128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-envoy\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.113169 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.113159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-workload-socket\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.213583 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.213545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-token\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.213775 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.213613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-envoy\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.213775 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.213644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-workload-socket\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.213775 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.213707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-workload-certs\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.213775 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.213734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnfr4\" (UniqueName: \"kubernetes.io/projected/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-kube-api-access-cnfr4\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.213775 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.213760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-credential-socket\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.214085 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.213781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-data\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.214085 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.213869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-podinfo\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.214085 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.213898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istiod-ca-cert\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.214192 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.214100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-workload-socket\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.214330 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.214307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-data\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.214401 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.214323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-workload-certs\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.214471 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.214454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-credential-socket\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.214804 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.214783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istiod-ca-cert\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.215924 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.215899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-envoy\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.216415 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.216398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-podinfo\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.221851 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.221822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-token\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.222322 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.222297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnfr4\" (UniqueName: \"kubernetes.io/projected/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-kube-api-access-cnfr4\") pod \"e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") " pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.323340 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.323248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:39.501933 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.501889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:39.604861 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.604834    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"]
Sep 09 20:42:39.607440 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:42:39.607407    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb7030d_c767_4bbc_9a28_171a14d5c7ec.slice/crio-ea8f3f0329b87ffa11e821bfa389e7e993692539ebeeb1b076032a32357ca2f9 WatchSource:0}: Error finding container ea8f3f0329b87ffa11e821bfa389e7e993692539ebeeb1b076032a32357ca2f9: Status 404 returned error can't find the container with id ea8f3f0329b87ffa11e821bfa389e7e993692539ebeeb1b076032a32357ca2f9
Sep 09 20:42:39.609678 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.609645    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:42:39.609773 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.609711    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:42:39.609773 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:39.609743    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:42:40.527861 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:40.527825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" event={"ID":"bfb7030d-c767-4bbc-9a28-171a14d5c7ec","Type":"ContainerStarted","Data":"ff115c0b2a665d65eb061018bc8c47b498e40ccaaf26ab8a35d6ceee810362f2"}
Sep 09 20:42:40.528315 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:40.527885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" event={"ID":"bfb7030d-c767-4bbc-9a28-171a14d5c7ec","Type":"ContainerStarted","Data":"ea8f3f0329b87ffa11e821bfa389e7e993692539ebeeb1b076032a32357ca2f9"}
Sep 09 20:42:40.549100 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:40.549029    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" podStartSLOduration=2.549005798 podStartE2EDuration="2.549005798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:42:40.546412208 +0000 UTC m=+1567.785514508" watchObservedRunningTime="2026-09-09 20:42:40.549005798 +0000 UTC m=+1567.788108100"
Sep 09 20:42:41.324269 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:41.324237    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:41.329314 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:41.329290    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:41.532253 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:41.532221    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:41.533365 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:41.533342    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:42:41.787312 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:41.787279    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:42.560639 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:42.560602    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:42.659326 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:42.659289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:43.202219 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:43.202181    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:42:43.373236 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:43.373192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:44.258103 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:44.258054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:44.356287 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:44.356250    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:44.773774 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:44.773739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:44.871283 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:44.871243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:44.914463 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:44.914424    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"]
Sep 09 20:42:44.917971 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:44.917947    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"
Sep 09 20:42:44.920875 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:44.920857    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-868e40-dockercfg-lc668\""
Sep 09 20:42:44.921005 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:44.920877    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-868e40\""
Sep 09 20:42:44.929614 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:44.929591    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"]
Sep 09 20:42:44.949820 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:44.949778    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:45.069945 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:45.069859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-678bk\" (UniqueName: \"kubernetes.io/projected/939047db-77d3-4306-bd4a-0449f1408193-kube-api-access-678bk\") pod \"payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7\" (UID: \"939047db-77d3-4306-bd4a-0449f1408193\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"
Sep 09 20:42:45.069945 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:45.069902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939047db-77d3-4306-bd4a-0449f1408193-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7\" (UID: \"939047db-77d3-4306-bd4a-0449f1408193\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"
Sep 09 20:42:45.170816 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:45.170778    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-678bk\" (UniqueName: \"kubernetes.io/projected/939047db-77d3-4306-bd4a-0449f1408193-kube-api-access-678bk\") pod \"payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7\" (UID: \"939047db-77d3-4306-bd4a-0449f1408193\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"
Sep 09 20:42:45.170816 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:45.170821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939047db-77d3-4306-bd4a-0449f1408193-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7\" (UID: \"939047db-77d3-4306-bd4a-0449f1408193\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"
Sep 09 20:42:45.171512 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:45.171492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939047db-77d3-4306-bd4a-0449f1408193-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7\" (UID: \"939047db-77d3-4306-bd4a-0449f1408193\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"
Sep 09 20:42:45.178424 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:45.178392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-678bk\" (UniqueName: \"kubernetes.io/projected/939047db-77d3-4306-bd4a-0449f1408193-kube-api-access-678bk\") pod \"payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7\" (UID: \"939047db-77d3-4306-bd4a-0449f1408193\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"
Sep 09 20:42:45.228700 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:45.228663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"
Sep 09 20:42:45.504241 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:45.504204    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"]
Sep 09 20:42:45.507623 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:42:45.507590    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939047db_77d3_4306_bd4a_0449f1408193.slice/crio-9c6283b112386a7a658451e891cca38f51cb8a9c3daf666e87e659c0af6205f4 WatchSource:0}: Error finding container 9c6283b112386a7a658451e891cca38f51cb8a9c3daf666e87e659c0af6205f4: Status 404 returned error can't find the container with id 9c6283b112386a7a658451e891cca38f51cb8a9c3daf666e87e659c0af6205f4
Sep 09 20:42:45.551759 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:45.551726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7" event={"ID":"939047db-77d3-4306-bd4a-0449f1408193","Type":"ContainerStarted","Data":"9c6283b112386a7a658451e891cca38f51cb8a9c3daf666e87e659c0af6205f4"}
Sep 09 20:42:46.558035 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:46.557997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7" event={"ID":"939047db-77d3-4306-bd4a-0449f1408193","Type":"ContainerStarted","Data":"384baf54fccc4ea2131048ba86544c37744346c72eed8238fa6fba1801b99b67"}
Sep 09 20:42:46.558650 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:46.558190    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"
Sep 09 20:42:46.576130 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:46.576080    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7" podStartSLOduration=2.5760655310000002 podStartE2EDuration="2.576065531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:42:46.574180985 +0000 UTC m=+1573.813283286" watchObservedRunningTime="2026-09-09 20:42:46.576065531 +0000 UTC m=+1573.815167828"
Sep 09 20:42:52.441019 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.440955    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87" podUID="f0147a28-33b1-4b27-a071-1bf4572db820" containerName="main" containerID="cri-o://89e889d10e1e1e94517c4d9f5d5362019a2536cbb8a610f8f3db5f42e1541eab" gracePeriod=45
Sep 09 20:42:52.588053 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.587990    2563 generic.go:356] "Generic (PLEG): container finished" podID="f0147a28-33b1-4b27-a071-1bf4572db820" containerID="89e889d10e1e1e94517c4d9f5d5362019a2536cbb8a610f8f3db5f42e1541eab" exitCode=0
Sep 09 20:42:52.588053 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.588013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87" event={"ID":"f0147a28-33b1-4b27-a071-1bf4572db820","Type":"ContainerDied","Data":"89e889d10e1e1e94517c4d9f5d5362019a2536cbb8a610f8f3db5f42e1541eab"}
Sep 09 20:42:52.726000 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.725970    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:42:52.846953 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.846903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-dshm\") pod \"f0147a28-33b1-4b27-a071-1bf4572db820\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") "
Sep 09 20:42:52.847274 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.847249    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-home\") pod \"f0147a28-33b1-4b27-a071-1bf4572db820\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") "
Sep 09 20:42:52.847480 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.847463    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-model-cache\") pod \"f0147a28-33b1-4b27-a071-1bf4572db820\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") "
Sep 09 20:42:52.847654 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.847639    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-tmp-dir\") pod \"f0147a28-33b1-4b27-a071-1bf4572db820\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") "
Sep 09 20:42:52.847812 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.847797    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0147a28-33b1-4b27-a071-1bf4572db820-tls-certs\") pod \"f0147a28-33b1-4b27-a071-1bf4572db820\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") "
Sep 09 20:42:52.847942 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.847927    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbkmp\" (UniqueName: \"kubernetes.io/projected/f0147a28-33b1-4b27-a071-1bf4572db820-kube-api-access-bbkmp\") pod \"f0147a28-33b1-4b27-a071-1bf4572db820\" (UID: \"f0147a28-33b1-4b27-a071-1bf4572db820\") "
Sep 09 20:42:52.851118 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.851095    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0147a28-33b1-4b27-a071-1bf4572db820-kube-api-access-bbkmp" (OuterVolumeSpecName: "kube-api-access-bbkmp") pod "f0147a28-33b1-4b27-a071-1bf4572db820" (UID: "f0147a28-33b1-4b27-a071-1bf4572db820"). InnerVolumeSpecName "kube-api-access-bbkmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:42:52.853842 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.853817    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-home" (OuterVolumeSpecName: "home") pod "f0147a28-33b1-4b27-a071-1bf4572db820" (UID: "f0147a28-33b1-4b27-a071-1bf4572db820"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:52.854176 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.854153    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0147a28-33b1-4b27-a071-1bf4572db820" (UID: "f0147a28-33b1-4b27-a071-1bf4572db820"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:52.854771 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.854747    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0147a28-33b1-4b27-a071-1bf4572db820" (UID: "f0147a28-33b1-4b27-a071-1bf4572db820"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:52.858528 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.858507    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-dshm" (OuterVolumeSpecName: "dshm") pod "f0147a28-33b1-4b27-a071-1bf4572db820" (UID: "f0147a28-33b1-4b27-a071-1bf4572db820"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:52.863900 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.863872    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0147a28-33b1-4b27-a071-1bf4572db820-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0147a28-33b1-4b27-a071-1bf4572db820" (UID: "f0147a28-33b1-4b27-a071-1bf4572db820"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:42:52.951384 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.951341    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbkmp\" (UniqueName: \"kubernetes.io/projected/f0147a28-33b1-4b27-a071-1bf4572db820-kube-api-access-bbkmp\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:42:52.951641 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.951621    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-dshm\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:42:52.951750 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.951738    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-home\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:42:52.951847 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.951833    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-model-cache\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:42:52.951931 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.951921    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0147a28-33b1-4b27-a071-1bf4572db820-tmp-dir\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:42:52.952015 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:52.952005    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0147a28-33b1-4b27-a071-1bf4572db820-tls-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:42:53.594308 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:53.594215    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"
Sep 09 20:42:53.594308 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:53.594258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87" event={"ID":"f0147a28-33b1-4b27-a071-1bf4572db820","Type":"ContainerDied","Data":"f94d868b37c43ee74e764c9ffb08e78ff13a3da91807090daf15cb4bb39f87e7"}
Sep 09 20:42:53.594856 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:53.594309    2563 scope.go:117] "RemoveContainer" containerID="89e889d10e1e1e94517c4d9f5d5362019a2536cbb8a610f8f3db5f42e1541eab"
Sep 09 20:42:53.612184 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:53.612158    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"]
Sep 09 20:42:53.614031 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:53.614007    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/rate-test-model-166b94-kserve-688768848c-6tk87"]
Sep 09 20:42:55.319243 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:55.319198    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0147a28-33b1-4b27-a071-1bf4572db820" path="/var/lib/kubelet/pods/f0147a28-33b1-4b27-a071-1bf4572db820/volumes"
Sep 09 20:42:55.509787 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:55.509748    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:42:57.565672 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:57.565641    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"
Sep 09 20:42:57.573709 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:57.573685    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 20:42:57.573825 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:57.573731    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:42:59.109113 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:59.109024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:59.222568 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:59.222527    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:59.625658 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:59.625623    2563 generic.go:356] "Generic (PLEG): container finished" podID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerID="ff115c0b2a665d65eb061018bc8c47b498e40ccaaf26ab8a35d6ceee810362f2" exitCode=0
Sep 09 20:42:59.625859 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:59.625701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" event={"ID":"bfb7030d-c767-4bbc-9a28-171a14d5c7ec","Type":"ContainerDied","Data":"ff115c0b2a665d65eb061018bc8c47b498e40ccaaf26ab8a35d6ceee810362f2"}
Sep 09 20:42:59.626139 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:59.626122    2563 scope.go:117] "RemoveContainer" containerID="ff115c0b2a665d65eb061018bc8c47b498e40ccaaf26ab8a35d6ceee810362f2"
Sep 09 20:42:59.627210 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:59.627184    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:42:59.627287 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:59.627268    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:42:59.627330 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:42:59.627306    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:43:00.632522 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:00.632485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" event={"ID":"bfb7030d-c767-4bbc-9a28-171a14d5c7ec","Type":"ContainerStarted","Data":"23825b66d7ed606a30ac4c9a3825647629de142d3de7654ab9d795fc3fa46c2f"}
Sep 09 20:43:01.324192 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:01.324157    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:43:01.324449 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:01.324429    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 09 20:43:01.324501 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:01.324478    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 09 20:43:02.324231 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:02.324192    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 09 20:43:02.324661 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:02.324293    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 09 20:43:03.323761 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:03.323720    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 09 20:43:03.323928 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:03.323775    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 09 20:43:04.323889 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:04.323854    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 09 20:43:04.324296 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:04.323920    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 09 20:43:05.324478 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:05.324449    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 09 20:43:05.324916 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:05.324500    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 09 20:43:06.485229 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:06.485197    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:43:06.485656 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:06.485467    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:43:06.486403 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:06.486385    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:43:07.181069 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:07.181032    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:07.486515 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:07.486476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:07.890155 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:07.890056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:08.210854 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.210815    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:08.550263 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.550177    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"]
Sep 09 20:43:08.550731 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.550508    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0147a28-33b1-4b27-a071-1bf4572db820" containerName="main"
Sep 09 20:43:08.550731 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.550522    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0147a28-33b1-4b27-a071-1bf4572db820" containerName="main"
Sep 09 20:43:08.550731 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.550608    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0147a28-33b1-4b27-a071-1bf4572db820" containerName="main"
Sep 09 20:43:08.553539 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.553517    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.555808 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.555785    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-697f19-openshift-default-dockercfg-8m979\""
Sep 09 20:43:08.565770 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.565748    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"]
Sep 09 20:43:08.584036 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.583994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:08.598218 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.598184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-workload-certs\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.598366 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.598222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-istio-data\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.598366 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.598242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtjdg\" (UniqueName: \"kubernetes.io/projected/f0732bd8-480d-442f-be41-f4363ad21c35-kube-api-access-vtjdg\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.598366 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.598261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-credential-socket\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.598366 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.598346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0732bd8-480d-442f-be41-f4363ad21c35-istio-token\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.598527 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.598387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0732bd8-480d-442f-be41-f4363ad21c35-istio-podinfo\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.598527 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.598425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-istio-envoy\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.598527 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.598469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0732bd8-480d-442f-be41-f4363ad21c35-istiod-ca-cert\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.598647 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.598538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-workload-socket\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.699534 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.699498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-workload-socket\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.699734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.699588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-workload-certs\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.699734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.699625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-istio-data\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.699734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.699650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjdg\" (UniqueName: \"kubernetes.io/projected/f0732bd8-480d-442f-be41-f4363ad21c35-kube-api-access-vtjdg\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.699734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.699680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-credential-socket\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.699734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.699720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0732bd8-480d-442f-be41-f4363ad21c35-istio-token\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.700028 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.699747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0732bd8-480d-442f-be41-f4363ad21c35-istio-podinfo\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.700028 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.699892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-istio-envoy\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.700028 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.699910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-workload-socket\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.700207 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.699994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-istio-data\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.700207 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.700000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0732bd8-480d-442f-be41-f4363ad21c35-istiod-ca-cert\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.700207 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.700042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-credential-socket\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.700207 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.700143    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-workload-certs\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.700614 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.700560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0732bd8-480d-442f-be41-f4363ad21c35-istiod-ca-cert\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.702153 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.702123    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-istio-envoy\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.702246 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.702215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0732bd8-480d-442f-be41-f4363ad21c35-istio-podinfo\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.711061 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.711034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0732bd8-480d-442f-be41-f4363ad21c35-istio-token\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.711283 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.711258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtjdg\" (UniqueName: \"kubernetes.io/projected/f0732bd8-480d-442f-be41-f4363ad21c35-kube-api-access-vtjdg\") pod \"e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") " pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:08.713251 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.713226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:08.828750 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.828658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:08.865674 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:08.865636    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:09.141326 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:09.141248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"]
Sep 09 20:43:09.144252 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:43:09.144210    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0732bd8_480d_442f_be41_f4363ad21c35.slice/crio-71f0374c3fdbc8ce1901c7c9e740f9f7d7db415d6c5b0683cb3bc18bf88dccf6 WatchSource:0}: Error finding container 71f0374c3fdbc8ce1901c7c9e740f9f7d7db415d6c5b0683cb3bc18bf88dccf6: Status 404 returned error can't find the container with id 71f0374c3fdbc8ce1901c7c9e740f9f7d7db415d6c5b0683cb3bc18bf88dccf6
Sep 09 20:43:09.146495 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:09.146463    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:43:09.146615 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:09.146544    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:43:09.146615 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:09.146605    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:43:09.486531 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:09.486480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:09.681458 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:09.681422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" event={"ID":"f0732bd8-480d-442f-be41-f4363ad21c35","Type":"ContainerStarted","Data":"d109952bce8076364e3a10526f3a0ad055a91ef1722991546c7c5bef10b00a94"}
Sep 09 20:43:09.681886 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:09.681486    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" event={"ID":"f0732bd8-480d-442f-be41-f4363ad21c35","Type":"ContainerStarted","Data":"71f0374c3fdbc8ce1901c7c9e740f9f7d7db415d6c5b0683cb3bc18bf88dccf6"}
Sep 09 20:43:09.704404 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:09.704342    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" podStartSLOduration=1.7043264059999998 podStartE2EDuration="1.704326406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:43:09.702774531 +0000 UTC m=+1596.941876832" watchObservedRunningTime="2026-09-09 20:43:09.704326406 +0000 UTC m=+1596.943428706"
Sep 09 20:43:09.866242 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:09.866155    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:10.871756 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:10.871722    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:10.979866 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:10.979832    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:11.654291 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:11.654248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:11.690047 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:11.690014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:11.690940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:11.690918    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:11.751258 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:11.751224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:12.580464 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:12.580365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:13.365674 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:13.365630    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:14.174714 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:14.174681    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:14.275958 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:14.275922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:14.535997 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:14.535963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:14.641833 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:14.641784    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:14.723091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:14.723052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:19.699492 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:19.699456    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:43:27.731745 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:27.731710    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 20:43:27.732115 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:27.731774    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:43:29.785593 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:29.785539    2563 generic.go:356] "Generic (PLEG): container finished" podID="f0732bd8-480d-442f-be41-f4363ad21c35" containerID="d109952bce8076364e3a10526f3a0ad055a91ef1722991546c7c5bef10b00a94" exitCode=0
Sep 09 20:43:29.786010 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:29.785621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" event={"ID":"f0732bd8-480d-442f-be41-f4363ad21c35","Type":"ContainerDied","Data":"d109952bce8076364e3a10526f3a0ad055a91ef1722991546c7c5bef10b00a94"}
Sep 09 20:43:29.786057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:29.786017    2563 scope.go:117] "RemoveContainer" containerID="d109952bce8076364e3a10526f3a0ad055a91ef1722991546c7c5bef10b00a94"
Sep 09 20:43:29.787072 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:29.787046    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:43:29.787127 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:29.787119    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:43:29.788263 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:29.787729    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:43:30.792192 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:30.792156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" event={"ID":"f0732bd8-480d-442f-be41-f4363ad21c35","Type":"ContainerStarted","Data":"89916c5b2442a52d7be518ce3d1b446cfacc31c9b64eca11d25ff62470a007d1"}
Sep 09 20:43:30.866588 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:30.866528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:30.866788 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:30.866686    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 09 20:43:30.866788 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:30.866745    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 09 20:43:31.866565 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:31.866526    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 09 20:43:31.867196 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:31.867159    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 09 20:43:32.867375 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:32.866922    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 09 20:43:32.867375 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:32.866988    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 09 20:43:33.866736 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:33.866631    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 09 20:43:33.866736 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:33.866703    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 09 20:43:34.867092 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:34.867050    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 09 20:43:34.867712 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:34.867665    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 09 20:43:35.866370 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:35.866326    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 09 20:43:35.866740 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:35.866704    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 09 20:43:36.287141 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:36.287106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:36.675273 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:36.675183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:36.869921 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:36.869889    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:36.870215 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:36.870197    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:36.870851 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:36.870834    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:43:36.986241 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:36.986206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:37.629855 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:37.629811    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:38.344763 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:38.344717    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:38.986860 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:38.986824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:43.048964 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:43.048900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:47.000695 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:43:47.000657    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:44:01.107310 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:01.107269    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:44:07.080151 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.080114    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7_939047db-77d3-4306-bd4a-0449f1408193/payload-processing/0.log"
Sep 09 20:44:07.592321 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.592285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:07.866550 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.866458    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"]
Sep 09 20:44:07.870282 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.870262    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:07.873034 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.873011    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-dd5d0a-openshift-default-dockercfg-lv6dz\""
Sep 09 20:44:07.880811 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.880786    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"]
Sep 09 20:44:07.902776 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.902741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-istio-envoy\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:07.902953 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.902790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-istio-data\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:07.902953 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.902820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77198f70-b61f-47c0-a791-82efecf98205-istio-token\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:07.902953 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.902845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx4wj\" (UniqueName: \"kubernetes.io/projected/77198f70-b61f-47c0-a791-82efecf98205-kube-api-access-zx4wj\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:07.902953 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.902873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-workload-socket\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:07.902953 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.902942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-credential-socket\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:07.903179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.903018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77198f70-b61f-47c0-a791-82efecf98205-istio-podinfo\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:07.903179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.903063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77198f70-b61f-47c0-a791-82efecf98205-istiod-ca-cert\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:07.903179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.903124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-workload-certs\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:07.964278 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:07.963787    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:08.003942 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.003905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-istio-envoy\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.003942 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.003946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-istio-data\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.004217 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.003965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77198f70-b61f-47c0-a791-82efecf98205-istio-token\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.004217 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.003984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx4wj\" (UniqueName: \"kubernetes.io/projected/77198f70-b61f-47c0-a791-82efecf98205-kube-api-access-zx4wj\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.004217 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.004011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-workload-socket\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.004217 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.004040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-credential-socket\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.004217 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.004068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77198f70-b61f-47c0-a791-82efecf98205-istio-podinfo\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.004217 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.004098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77198f70-b61f-47c0-a791-82efecf98205-istiod-ca-cert\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.004217 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.004159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-workload-certs\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.004682 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.004423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-istio-data\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.004682 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.004465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-workload-socket\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.004682 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.004503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-workload-certs\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.004852 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.004758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-credential-socket\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.005059 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.005036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77198f70-b61f-47c0-a791-82efecf98205-istiod-ca-cert\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.007073 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.007043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-istio-envoy\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.007302 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.007272    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77198f70-b61f-47c0-a791-82efecf98205-istio-podinfo\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.012010 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.011980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx4wj\" (UniqueName: \"kubernetes.io/projected/77198f70-b61f-47c0-a791-82efecf98205-kube-api-access-zx4wj\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.012091 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.012005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77198f70-b61f-47c0-a791-82efecf98205-istio-token\") pod \"e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") " pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.181717 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.181618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:08.400094 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.400051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:08.459381 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.459343    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"]
Sep 09 20:44:08.461399 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:44:08.461321    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77198f70_b61f_47c0_a791_82efecf98205.slice/crio-a59938e0f4fe356bb4c7df335fa29985912ad03d5f29dd96978794934eabf993 WatchSource:0}: Error finding container a59938e0f4fe356bb4c7df335fa29985912ad03d5f29dd96978794934eabf993: Status 404 returned error can't find the container with id a59938e0f4fe356bb4c7df335fa29985912ad03d5f29dd96978794934eabf993
Sep 09 20:44:08.465819 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.465783    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:44:08.466003 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.465871    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:44:08.466003 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.465919    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:44:08.974064 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.974026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" event={"ID":"77198f70-b61f-47c0-a791-82efecf98205","Type":"ContainerStarted","Data":"a7d07675034d242595310ab9ee2d5c1b07188418c5b3ab0ca0fa35032098a5da"}
Sep 09 20:44:08.974240 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.974089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" event={"ID":"77198f70-b61f-47c0-a791-82efecf98205","Type":"ContainerStarted","Data":"a59938e0f4fe356bb4c7df335fa29985912ad03d5f29dd96978794934eabf993"}
Sep 09 20:44:08.994846 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:08.994789    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" podStartSLOduration=1.9947699989999998 podStartE2EDuration="1.994769999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:44:08.994019335 +0000 UTC m=+1656.233121635" watchObservedRunningTime="2026-09-09 20:44:08.994769999 +0000 UTC m=+1656.233872301"
Sep 09 20:44:09.182444 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:09.182402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:09.188460 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:09.188432    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:09.979002 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:09.978959    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:09.980054 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:09.980029    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:10.632162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:10.632131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:10.912824 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:10.912726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:11.142649 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:11.142609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:11.781247 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:11.781204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:12.569281 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:12.569174    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:13.237289 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:13.237236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:14.078649 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:14.078612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:14.179823 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:14.179780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:14.643631 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:14.643596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:14.742003 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:14.741962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:14.980921 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:14.980885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:21.117226 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:21.117191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:44:32.097400 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:32.097365    2563 generic.go:356] "Generic (PLEG): container finished" podID="77198f70-b61f-47c0-a791-82efecf98205" containerID="a7d07675034d242595310ab9ee2d5c1b07188418c5b3ab0ca0fa35032098a5da" exitCode=0
Sep 09 20:44:32.097831 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:32.097432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" event={"ID":"77198f70-b61f-47c0-a791-82efecf98205","Type":"ContainerDied","Data":"a7d07675034d242595310ab9ee2d5c1b07188418c5b3ab0ca0fa35032098a5da"}
Sep 09 20:44:32.097882 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:32.097857    2563 scope.go:117] "RemoveContainer" containerID="a7d07675034d242595310ab9ee2d5c1b07188418c5b3ab0ca0fa35032098a5da"
Sep 09 20:44:32.098981 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:32.098953    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:44:32.099045 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:32.099022    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:44:32.099082 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:32.099059    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:44:33.103264 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:33.103230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" event={"ID":"77198f70-b61f-47c0-a791-82efecf98205","Type":"ContainerStarted","Data":"58d4b56b556b113e2535883ac5fbbf0ad1df5d00bc500a8be1a9808a7a0eafbb"}
Sep 09 20:44:33.181968 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:33.181935    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:33.182188 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:33.182166    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 09 20:44:33.182241 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:33.182214    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" podUID="77198f70-b61f-47c0-a791-82efecf98205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 09 20:44:34.182383 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:34.182344    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 09 20:44:34.182801 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:34.182407    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" podUID="77198f70-b61f-47c0-a791-82efecf98205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 09 20:44:35.182758 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:35.182719    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 09 20:44:35.183126 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:35.182779    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" podUID="77198f70-b61f-47c0-a791-82efecf98205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 09 20:44:36.182591 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:36.182534    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 09 20:44:36.182786 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:36.182612    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" podUID="77198f70-b61f-47c0-a791-82efecf98205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 09 20:44:37.107774 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:37.107733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:37.182735 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:37.182700    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 09 20:44:37.182907 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:37.182754    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" podUID="77198f70-b61f-47c0-a791-82efecf98205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 09 20:44:37.878952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:37.878914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:38.182180 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:38.182086    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 09 20:44:38.182180 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:38.182147    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" podUID="77198f70-b61f-47c0-a791-82efecf98205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 09 20:44:38.675231 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:38.675189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:39.186227 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:39.186195    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:39.186604 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:39.186564    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:39.187269 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:39.187253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:44:39.377202 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:39.377162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:40.047346 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:40.047309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:40.694896 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:40.694852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:40.958512 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:40.958471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:41.625993 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:41.625962    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5"]
Sep 09 20:44:41.629299 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:41.629280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5"
Sep 09 20:44:41.631770 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:41.631747    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:44:41.631911 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:41.631747    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:44:41.636322 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:41.636291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5"]
Sep 09 20:44:41.726439 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:41.726402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af382183-d25b-4def-b4f6-c8e137c44f0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5\" (UID: \"af382183-d25b-4def-b4f6-c8e137c44f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5"
Sep 09 20:44:41.827550 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:41.827509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af382183-d25b-4def-b4f6-c8e137c44f0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5\" (UID: \"af382183-d25b-4def-b4f6-c8e137c44f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5"
Sep 09 20:44:41.828171 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:41.828145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af382183-d25b-4def-b4f6-c8e137c44f0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5\" (UID: \"af382183-d25b-4def-b4f6-c8e137c44f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5"
Sep 09 20:44:41.940331 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:41.940231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5"
Sep 09 20:44:42.239694 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:42.239662    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5"]
Sep 09 20:44:42.242812 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:44:42.242782    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf382183_d25b_4def_b4f6_c8e137c44f0b.slice/crio-8da03eae8bce78b71ebee2d72c6ac25a11bdbb5b9143e5afd8978f9f984ec745 WatchSource:0}: Error finding container 8da03eae8bce78b71ebee2d72c6ac25a11bdbb5b9143e5afd8978f9f984ec745: Status 404 returned error can't find the container with id 8da03eae8bce78b71ebee2d72c6ac25a11bdbb5b9143e5afd8978f9f984ec745
Sep 09 20:44:42.685742 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:42.685703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:43.148532 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:43.148495    2563 generic.go:356] "Generic (PLEG): container finished" podID="af382183-d25b-4def-b4f6-c8e137c44f0b" containerID="2f27466902f85eb275aed9f618d7c2e98d005c1379358547e7037fe3a96c9024" exitCode=0
Sep 09 20:44:43.148701 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:43.148602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5" event={"ID":"af382183-d25b-4def-b4f6-c8e137c44f0b","Type":"ContainerDied","Data":"2f27466902f85eb275aed9f618d7c2e98d005c1379358547e7037fe3a96c9024"}
Sep 09 20:44:43.148701 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:43.148680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5" event={"ID":"af382183-d25b-4def-b4f6-c8e137c44f0b","Type":"ContainerStarted","Data":"8da03eae8bce78b71ebee2d72c6ac25a11bdbb5b9143e5afd8978f9f984ec745"}
Sep 09 20:44:43.359175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:43.359136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:44.017388 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:44.017349    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:44.282861 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:44.282834    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5"
Sep 09 20:44:44.450099 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:44.450057    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af382183-d25b-4def-b4f6-c8e137c44f0b-maas-api-service-ca\") pod \"af382183-d25b-4def-b4f6-c8e137c44f0b\" (UID: \"af382183-d25b-4def-b4f6-c8e137c44f0b\") "
Sep 09 20:44:44.450449 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:44.450428    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af382183-d25b-4def-b4f6-c8e137c44f0b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af382183-d25b-4def-b4f6-c8e137c44f0b" (UID: "af382183-d25b-4def-b4f6-c8e137c44f0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:44:44.551302 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:44.551194    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af382183-d25b-4def-b4f6-c8e137c44f0b-maas-api-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:44:44.787208 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:44.787170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:45.158646 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:45.158617    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5"
Sep 09 20:44:45.159125 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:45.158645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5" event={"ID":"af382183-d25b-4def-b4f6-c8e137c44f0b","Type":"ContainerDied","Data":"8da03eae8bce78b71ebee2d72c6ac25a11bdbb5b9143e5afd8978f9f984ec745"}
Sep 09 20:44:45.159125 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:45.158694    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8da03eae8bce78b71ebee2d72c6ac25a11bdbb5b9143e5afd8978f9f984ec745"
Sep 09 20:44:45.464931 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:45.464893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:46.143519 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:46.143468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:52.646716 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:52.646667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:52.750031 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:52.749999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:52.847203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:52.847167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:52.858064 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:52.858027    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5"]
Sep 09 20:44:52.861986 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:52.861959    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dd5d0a-n5hs5"]
Sep 09 20:44:52.949628 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:52.949519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:53.226757 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:53.226721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:53.314336 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:53.314296    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af382183-d25b-4def-b4f6-c8e137c44f0b" path="/var/lib/kubelet/pods/af382183-d25b-4def-b4f6-c8e137c44f0b/volumes"
Sep 09 20:44:53.322374 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:53.322345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:54.090588 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:54.090536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:54.999926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:54.999837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:55.840376 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:55.840334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:56.519328 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:56.519283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:57.201350 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:57.201310    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:44:59.081678 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:59.081635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:59.185079 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:59.185041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:59.337834 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:59.337746    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"]
Sep 09 20:44:59.338020 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:59.337991    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" podUID="77198f70-b61f-47c0-a791-82efecf98205" containerName="istio-proxy" containerID="cri-o://58d4b56b556b113e2535883ac5fbbf0ad1df5d00bc500a8be1a9808a7a0eafbb" gracePeriod=30
Sep 09 20:44:59.366634 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:44:59.366602    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:00.133094 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.133060    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs"]
Sep 09 20:45:00.133471 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.133406    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af382183-d25b-4def-b4f6-c8e137c44f0b" containerName="revoke-keys"
Sep 09 20:45:00.133471 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.133418    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="af382183-d25b-4def-b4f6-c8e137c44f0b" containerName="revoke-keys"
Sep 09 20:45:00.133544 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.133477    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="af382183-d25b-4def-b4f6-c8e137c44f0b" containerName="revoke-keys"
Sep 09 20:45:00.136726 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.136704    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs"
Sep 09 20:45:00.140039 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.140015    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:45:00.147011 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.145507    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816445-2c4jr"]
Sep 09 20:45:00.155149 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.155119    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d"]
Sep 09 20:45:00.155310 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.155202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-2c4jr"
Sep 09 20:45:00.158781 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.158755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs"
Sep 09 20:45:00.159300 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.159280    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs"]
Sep 09 20:45:00.159554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.159540    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d"]
Sep 09 20:45:00.159722 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.159495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d"
Sep 09 20:45:00.171346 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.171325    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-2c4jr"
Sep 09 20:45:00.177694 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.177524    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816445-2c4jr"]
Sep 09 20:45:00.211617 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.211480    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d"
Sep 09 20:45:00.441625 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.441600    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs"]
Sep 09 20:45:00.444056 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:45:00.444024    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeda97a1_1ac3_49bb_8ce9_a4d6f9016257.slice/crio-03734d06704a6fc389371fdf036774a2893138f659540619ee4858bf13538863 WatchSource:0}: Error finding container 03734d06704a6fc389371fdf036774a2893138f659540619ee4858bf13538863: Status 404 returned error can't find the container with id 03734d06704a6fc389371fdf036774a2893138f659540619ee4858bf13538863
Sep 09 20:45:00.468581 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.468404    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816445-2c4jr"]
Sep 09 20:45:00.475687 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:45:00.475661    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef979d8_f5ac_48e7_8216_10222766c6a4.slice/crio-12f3aab7a08b8821ec73af6d11ab5e453c6d1bb6b1d8e8592714564c41494082 WatchSource:0}: Error finding container 12f3aab7a08b8821ec73af6d11ab5e453c6d1bb6b1d8e8592714564c41494082: Status 404 returned error can't find the container with id 12f3aab7a08b8821ec73af6d11ab5e453c6d1bb6b1d8e8592714564c41494082
Sep 09 20:45:00.504543 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:00.504517    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d"]
Sep 09 20:45:00.506964 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:45:00.506932    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d0ab72_eb5a_4318_8df3_d1ad2b62497c.slice/crio-327cfd50a3ff4315f451b88786eb263c56ef85793f8fa0aa75e23a54ab3e549a WatchSource:0}: Error finding container 327cfd50a3ff4315f451b88786eb263c56ef85793f8fa0aa75e23a54ab3e549a: Status 404 returned error can't find the container with id 327cfd50a3ff4315f451b88786eb263c56ef85793f8fa0aa75e23a54ab3e549a
Sep 09 20:45:01.235025 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:01.234990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs" event={"ID":"aeda97a1-1ac3-49bb-8ce9-a4d6f9016257","Type":"ContainerStarted","Data":"65694cabed7d981787cb29038fdfdfaa1264b31cbdcc311f795c47bd0de91b01"}
Sep 09 20:45:01.235501 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:01.235056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs" event={"ID":"aeda97a1-1ac3-49bb-8ce9-a4d6f9016257","Type":"ContainerStarted","Data":"03734d06704a6fc389371fdf036774a2893138f659540619ee4858bf13538863"}
Sep 09 20:45:01.236594 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:01.236552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d" event={"ID":"06d0ab72-eb5a-4318-8df3-d1ad2b62497c","Type":"ContainerStarted","Data":"c6b51b230a2d3f9ab5c1a991dff424d8abaa36ec5d79d357151579a7e7d4eb66"}
Sep 09 20:45:01.236684 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:01.236624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d" event={"ID":"06d0ab72-eb5a-4318-8df3-d1ad2b62497c","Type":"ContainerStarted","Data":"327cfd50a3ff4315f451b88786eb263c56ef85793f8fa0aa75e23a54ab3e549a"}
Sep 09 20:45:01.238034 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:01.238005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-2c4jr" event={"ID":"4ef979d8-f5ac-48e7-8216-10222766c6a4","Type":"ContainerStarted","Data":"0dcffd8b4364488cea1a5226ac9b1aa63c0434c23487eb904c9f330d9aae27ea"}
Sep 09 20:45:01.238158 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:01.238042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-2c4jr" event={"ID":"4ef979d8-f5ac-48e7-8216-10222766c6a4","Type":"ContainerStarted","Data":"12f3aab7a08b8821ec73af6d11ab5e453c6d1bb6b1d8e8592714564c41494082"}
Sep 09 20:45:01.248897 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:01.248847    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs" podStartSLOduration=1.248827012 podStartE2EDuration="1.248827012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:45:01.247684643 +0000 UTC m=+1708.486786942" watchObservedRunningTime="2026-09-09 20:45:01.248827012 +0000 UTC m=+1708.487929313"
Sep 09 20:45:01.263220 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:01.263170    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-2c4jr" podStartSLOduration=1.263155087 podStartE2EDuration="1.263155087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:45:01.261330952 +0000 UTC m=+1708.500433253" watchObservedRunningTime="2026-09-09 20:45:01.263155087 +0000 UTC m=+1708.502257386"
Sep 09 20:45:01.278246 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:01.278172    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d" podStartSLOduration=1.2781541810000001 podStartE2EDuration="1.278154181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:45:01.27743555 +0000 UTC m=+1708.516537851" watchObservedRunningTime="2026-09-09 20:45:01.278154181 +0000 UTC m=+1708.517256526"
Sep 09 20:45:02.244330 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:02.244287    2563 generic.go:356] "Generic (PLEG): container finished" podID="aeda97a1-1ac3-49bb-8ce9-a4d6f9016257" containerID="65694cabed7d981787cb29038fdfdfaa1264b31cbdcc311f795c47bd0de91b01" exitCode=0
Sep 09 20:45:02.244831 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:02.244374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs" event={"ID":"aeda97a1-1ac3-49bb-8ce9-a4d6f9016257","Type":"ContainerDied","Data":"65694cabed7d981787cb29038fdfdfaa1264b31cbdcc311f795c47bd0de91b01"}
Sep 09 20:45:02.246175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:02.246149    2563 generic.go:356] "Generic (PLEG): container finished" podID="06d0ab72-eb5a-4318-8df3-d1ad2b62497c" containerID="c6b51b230a2d3f9ab5c1a991dff424d8abaa36ec5d79d357151579a7e7d4eb66" exitCode=0
Sep 09 20:45:02.246318 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:02.246225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d" event={"ID":"06d0ab72-eb5a-4318-8df3-d1ad2b62497c","Type":"ContainerDied","Data":"c6b51b230a2d3f9ab5c1a991dff424d8abaa36ec5d79d357151579a7e7d4eb66"}
Sep 09 20:45:02.247874 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:02.247852    2563 generic.go:356] "Generic (PLEG): container finished" podID="4ef979d8-f5ac-48e7-8216-10222766c6a4" containerID="0dcffd8b4364488cea1a5226ac9b1aa63c0434c23487eb904c9f330d9aae27ea" exitCode=0
Sep 09 20:45:02.247988 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:02.247893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-2c4jr" event={"ID":"4ef979d8-f5ac-48e7-8216-10222766c6a4","Type":"ContainerDied","Data":"0dcffd8b4364488cea1a5226ac9b1aa63c0434c23487eb904c9f330d9aae27ea"}
Sep 09 20:45:03.129361 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:03.129325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:03.406467 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:03.406435    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-2c4jr"
Sep 09 20:45:03.453559 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:03.453519    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d"
Sep 09 20:45:03.457069 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:03.457040    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs"
Sep 09 20:45:04.126472 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.126430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:04.258674 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.258646    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d"
Sep 09 20:45:04.258857 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.258641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d" event={"ID":"06d0ab72-eb5a-4318-8df3-d1ad2b62497c","Type":"ContainerDied","Data":"327cfd50a3ff4315f451b88786eb263c56ef85793f8fa0aa75e23a54ab3e549a"}
Sep 09 20:45:04.258857 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.258768    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="327cfd50a3ff4315f451b88786eb263c56ef85793f8fa0aa75e23a54ab3e549a"
Sep 09 20:45:04.259973 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.259951    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-2c4jr"
Sep 09 20:45:04.260101 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.259969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816445-2c4jr" event={"ID":"4ef979d8-f5ac-48e7-8216-10222766c6a4","Type":"ContainerDied","Data":"12f3aab7a08b8821ec73af6d11ab5e453c6d1bb6b1d8e8592714564c41494082"}
Sep 09 20:45:04.260101 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.260016    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12f3aab7a08b8821ec73af6d11ab5e453c6d1bb6b1d8e8592714564c41494082"
Sep 09 20:45:04.261295 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.261277    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs"
Sep 09 20:45:04.261404 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.261303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs" event={"ID":"aeda97a1-1ac3-49bb-8ce9-a4d6f9016257","Type":"ContainerDied","Data":"03734d06704a6fc389371fdf036774a2893138f659540619ee4858bf13538863"}
Sep 09 20:45:04.261404 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.261342    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03734d06704a6fc389371fdf036774a2893138f659540619ee4858bf13538863"
Sep 09 20:45:04.596716 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.596688    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:45:04.631096 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631059    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-workload-certs\") pod \"77198f70-b61f-47c0-a791-82efecf98205\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") "
Sep 09 20:45:04.631267 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-credential-socket\") pod \"77198f70-b61f-47c0-a791-82efecf98205\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") "
Sep 09 20:45:04.631267 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631160    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx4wj\" (UniqueName: \"kubernetes.io/projected/77198f70-b61f-47c0-a791-82efecf98205-kube-api-access-zx4wj\") pod \"77198f70-b61f-47c0-a791-82efecf98205\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") "
Sep 09 20:45:04.631267 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631185    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77198f70-b61f-47c0-a791-82efecf98205-istio-podinfo\") pod \"77198f70-b61f-47c0-a791-82efecf98205\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") "
Sep 09 20:45:04.631267 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631217    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-istio-envoy\") pod \"77198f70-b61f-47c0-a791-82efecf98205\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") "
Sep 09 20:45:04.631267 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631238    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77198f70-b61f-47c0-a791-82efecf98205-istio-token\") pod \"77198f70-b61f-47c0-a791-82efecf98205\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") "
Sep 09 20:45:04.631560 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631295    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-istio-data\") pod \"77198f70-b61f-47c0-a791-82efecf98205\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") "
Sep 09 20:45:04.631560 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631319    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-workload-socket\") pod \"77198f70-b61f-47c0-a791-82efecf98205\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") "
Sep 09 20:45:04.631560 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77198f70-b61f-47c0-a791-82efecf98205-istiod-ca-cert\") pod \"77198f70-b61f-47c0-a791-82efecf98205\" (UID: \"77198f70-b61f-47c0-a791-82efecf98205\") "
Sep 09 20:45:04.631560 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "77198f70-b61f-47c0-a791-82efecf98205" (UID: "77198f70-b61f-47c0-a791-82efecf98205"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:04.631560 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631507    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "77198f70-b61f-47c0-a791-82efecf98205" (UID: "77198f70-b61f-47c0-a791-82efecf98205"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:04.631936 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631883    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:45:04.631936 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631915    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:45:04.632086 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.631949    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "77198f70-b61f-47c0-a791-82efecf98205" (UID: "77198f70-b61f-47c0-a791-82efecf98205"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:04.632086 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.632019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77198f70-b61f-47c0-a791-82efecf98205-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "77198f70-b61f-47c0-a791-82efecf98205" (UID: "77198f70-b61f-47c0-a791-82efecf98205"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:45:04.632086 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.632056    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-istio-data" (OuterVolumeSpecName: "istio-data") pod "77198f70-b61f-47c0-a791-82efecf98205" (UID: "77198f70-b61f-47c0-a791-82efecf98205"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:04.633782 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.633754    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "77198f70-b61f-47c0-a791-82efecf98205" (UID: "77198f70-b61f-47c0-a791-82efecf98205"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:04.633897 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.633837    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77198f70-b61f-47c0-a791-82efecf98205-kube-api-access-zx4wj" (OuterVolumeSpecName: "kube-api-access-zx4wj") pod "77198f70-b61f-47c0-a791-82efecf98205" (UID: "77198f70-b61f-47c0-a791-82efecf98205"). InnerVolumeSpecName "kube-api-access-zx4wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:45:04.633998 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.633981    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77198f70-b61f-47c0-a791-82efecf98205-istio-token" (OuterVolumeSpecName: "istio-token") pod "77198f70-b61f-47c0-a791-82efecf98205" (UID: "77198f70-b61f-47c0-a791-82efecf98205"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:45:04.634092 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.634067    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77198f70-b61f-47c0-a791-82efecf98205-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "77198f70-b61f-47c0-a791-82efecf98205" (UID: "77198f70-b61f-47c0-a791-82efecf98205"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:45:04.732547 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.732508    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx4wj\" (UniqueName: \"kubernetes.io/projected/77198f70-b61f-47c0-a791-82efecf98205-kube-api-access-zx4wj\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:45:04.732547 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.732543    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77198f70-b61f-47c0-a791-82efecf98205-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:45:04.732547 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.732554    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:45:04.732805 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.732563    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77198f70-b61f-47c0-a791-82efecf98205-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:45:04.732805 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.732586    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:45:04.732805 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.732595    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77198f70-b61f-47c0-a791-82efecf98205-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:45:04.732805 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:04.732603    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77198f70-b61f-47c0-a791-82efecf98205-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:45:05.087525 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.087437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:05.268193 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.268157    2563 generic.go:356] "Generic (PLEG): container finished" podID="77198f70-b61f-47c0-a791-82efecf98205" containerID="58d4b56b556b113e2535883ac5fbbf0ad1df5d00bc500a8be1a9808a7a0eafbb" exitCode=0
Sep 09 20:45:05.268364 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.268203    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"
Sep 09 20:45:05.268364 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.268239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" event={"ID":"77198f70-b61f-47c0-a791-82efecf98205","Type":"ContainerDied","Data":"58d4b56b556b113e2535883ac5fbbf0ad1df5d00bc500a8be1a9808a7a0eafbb"}
Sep 09 20:45:05.268364 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.268296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m" event={"ID":"77198f70-b61f-47c0-a791-82efecf98205","Type":"ContainerDied","Data":"a59938e0f4fe356bb4c7df335fa29985912ad03d5f29dd96978794934eabf993"}
Sep 09 20:45:05.268364 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.268326    2563 scope.go:117] "RemoveContainer" containerID="58d4b56b556b113e2535883ac5fbbf0ad1df5d00bc500a8be1a9808a7a0eafbb"
Sep 09 20:45:05.277751 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.277732    2563 scope.go:117] "RemoveContainer" containerID="a7d07675034d242595310ab9ee2d5c1b07188418c5b3ab0ca0fa35032098a5da"
Sep 09 20:45:05.286750 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.286726    2563 scope.go:117] "RemoveContainer" containerID="58d4b56b556b113e2535883ac5fbbf0ad1df5d00bc500a8be1a9808a7a0eafbb"
Sep 09 20:45:05.287036 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:45:05.287017    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d4b56b556b113e2535883ac5fbbf0ad1df5d00bc500a8be1a9808a7a0eafbb\": container with ID starting with 58d4b56b556b113e2535883ac5fbbf0ad1df5d00bc500a8be1a9808a7a0eafbb not found: ID does not exist" containerID="58d4b56b556b113e2535883ac5fbbf0ad1df5d00bc500a8be1a9808a7a0eafbb"
Sep 09 20:45:05.287097 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.287046    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d4b56b556b113e2535883ac5fbbf0ad1df5d00bc500a8be1a9808a7a0eafbb"} err="failed to get container status \"58d4b56b556b113e2535883ac5fbbf0ad1df5d00bc500a8be1a9808a7a0eafbb\": rpc error: code = NotFound desc = could not find container \"58d4b56b556b113e2535883ac5fbbf0ad1df5d00bc500a8be1a9808a7a0eafbb\": container with ID starting with 58d4b56b556b113e2535883ac5fbbf0ad1df5d00bc500a8be1a9808a7a0eafbb not found: ID does not exist"
Sep 09 20:45:05.287097 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.287065    2563 scope.go:117] "RemoveContainer" containerID="a7d07675034d242595310ab9ee2d5c1b07188418c5b3ab0ca0fa35032098a5da"
Sep 09 20:45:05.287349 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:45:05.287327    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d07675034d242595310ab9ee2d5c1b07188418c5b3ab0ca0fa35032098a5da\": container with ID starting with a7d07675034d242595310ab9ee2d5c1b07188418c5b3ab0ca0fa35032098a5da not found: ID does not exist" containerID="a7d07675034d242595310ab9ee2d5c1b07188418c5b3ab0ca0fa35032098a5da"
Sep 09 20:45:05.287402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.287360    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d07675034d242595310ab9ee2d5c1b07188418c5b3ab0ca0fa35032098a5da"} err="failed to get container status \"a7d07675034d242595310ab9ee2d5c1b07188418c5b3ab0ca0fa35032098a5da\": rpc error: code = NotFound desc = could not find container \"a7d07675034d242595310ab9ee2d5c1b07188418c5b3ab0ca0fa35032098a5da\": container with ID starting with a7d07675034d242595310ab9ee2d5c1b07188418c5b3ab0ca0fa35032098a5da not found: ID does not exist"
Sep 09 20:45:05.294048 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.294022    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"]
Sep 09 20:45:05.300100 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.300072    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-dd5d0a-openshift-default-b6bfcc8f6-tw75m"]
Sep 09 20:45:05.312799 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.312768    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77198f70-b61f-47c0-a791-82efecf98205" path="/var/lib/kubelet/pods/77198f70-b61f-47c0-a791-82efecf98205/volumes"
Sep 09 20:45:05.815203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:05.815147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:06.468953 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:06.468918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:07.289941 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:07.289904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:16.497580 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:16.497545    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:45:18.473123 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.473069    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:18.483606 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.483554    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs"]
Sep 09 20:45:18.485112 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.485091    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-868e40-29816445-89njs"]
Sep 09 20:45:18.519080 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.519048    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"]
Sep 09 20:45:18.519300 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.519266    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7" podUID="939047db-77d3-4306-bd4a-0449f1408193" containerName="payload-processing" containerID="cri-o://384baf54fccc4ea2131048ba86544c37744346c72eed8238fa6fba1801b99b67" gracePeriod=30
Sep 09 20:45:18.570111 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.570075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:18.675057 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.675016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:18.748900 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.748812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:18.770600 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.770548    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"
Sep 09 20:45:18.855292 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.855250    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-678bk\" (UniqueName: \"kubernetes.io/projected/939047db-77d3-4306-bd4a-0449f1408193-kube-api-access-678bk\") pod \"939047db-77d3-4306-bd4a-0449f1408193\" (UID: \"939047db-77d3-4306-bd4a-0449f1408193\") "
Sep 09 20:45:18.855484 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.855311    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939047db-77d3-4306-bd4a-0449f1408193-plugins-config-volume\") pod \"939047db-77d3-4306-bd4a-0449f1408193\" (UID: \"939047db-77d3-4306-bd4a-0449f1408193\") "
Sep 09 20:45:18.855679 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.855650    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939047db-77d3-4306-bd4a-0449f1408193-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "939047db-77d3-4306-bd4a-0449f1408193" (UID: "939047db-77d3-4306-bd4a-0449f1408193"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:45:18.857505 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.857482    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939047db-77d3-4306-bd4a-0449f1408193-kube-api-access-678bk" (OuterVolumeSpecName: "kube-api-access-678bk") pod "939047db-77d3-4306-bd4a-0449f1408193" (UID: "939047db-77d3-4306-bd4a-0449f1408193"). InnerVolumeSpecName "kube-api-access-678bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:45:18.956123 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.956088    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-678bk\" (UniqueName: \"kubernetes.io/projected/939047db-77d3-4306-bd4a-0449f1408193-kube-api-access-678bk\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:45:18.956123 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:18.956119    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939047db-77d3-4306-bd4a-0449f1408193-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:45:19.092780 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:19.092671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:19.226402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:19.226301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:19.313189 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:19.313157    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeda97a1-1ac3-49bb-8ce9-a4d6f9016257" path="/var/lib/kubelet/pods/aeda97a1-1ac3-49bb-8ce9-a4d6f9016257/volumes"
Sep 09 20:45:19.342502 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:19.342468    2563 generic.go:356] "Generic (PLEG): container finished" podID="939047db-77d3-4306-bd4a-0449f1408193" containerID="384baf54fccc4ea2131048ba86544c37744346c72eed8238fa6fba1801b99b67" exitCode=0
Sep 09 20:45:19.342705 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:19.342523    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"
Sep 09 20:45:19.342705 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:19.342524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7" event={"ID":"939047db-77d3-4306-bd4a-0449f1408193","Type":"ContainerDied","Data":"384baf54fccc4ea2131048ba86544c37744346c72eed8238fa6fba1801b99b67"}
Sep 09 20:45:19.342705 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:19.342638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7" event={"ID":"939047db-77d3-4306-bd4a-0449f1408193","Type":"ContainerDied","Data":"9c6283b112386a7a658451e891cca38f51cb8a9c3daf666e87e659c0af6205f4"}
Sep 09 20:45:19.342705 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:19.342671    2563 scope.go:117] "RemoveContainer" containerID="384baf54fccc4ea2131048ba86544c37744346c72eed8238fa6fba1801b99b67"
Sep 09 20:45:19.350892 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:19.350867    2563 scope.go:117] "RemoveContainer" containerID="384baf54fccc4ea2131048ba86544c37744346c72eed8238fa6fba1801b99b67"
Sep 09 20:45:19.351181 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:45:19.351161    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384baf54fccc4ea2131048ba86544c37744346c72eed8238fa6fba1801b99b67\": container with ID starting with 384baf54fccc4ea2131048ba86544c37744346c72eed8238fa6fba1801b99b67 not found: ID does not exist" containerID="384baf54fccc4ea2131048ba86544c37744346c72eed8238fa6fba1801b99b67"
Sep 09 20:45:19.351254 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:19.351192    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"384baf54fccc4ea2131048ba86544c37744346c72eed8238fa6fba1801b99b67"} err="failed to get container status \"384baf54fccc4ea2131048ba86544c37744346c72eed8238fa6fba1801b99b67\": rpc error: code = NotFound desc = could not find container \"384baf54fccc4ea2131048ba86544c37744346c72eed8238fa6fba1801b99b67\": container with ID starting with 384baf54fccc4ea2131048ba86544c37744346c72eed8238fa6fba1801b99b67 not found: ID does not exist"
Sep 09 20:45:19.376695 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:19.376657    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"]
Sep 09 20:45:19.401255 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:19.401228    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-868e40-6566bf5479-xdsf7"]
Sep 09 20:45:20.156866 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:20.156829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:20.975476 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:20.975433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:21.313332 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:21.313250    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="939047db-77d3-4306-bd4a-0449f1408193" path="/var/lib/kubelet/pods/939047db-77d3-4306-bd4a-0449f1408193/volumes"
Sep 09 20:45:21.723440 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:21.723401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:24.110672 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:24.110637    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:28.307372 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:45:28.307338    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:46:00.839455 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:00.839417    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:02.270781 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:02.270745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:08.520307 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:08.520270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:08.718849 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:08.718802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:08.846242 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:08.846150    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"]
Sep 09 20:46:08.846436 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:08.846412    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerName="istio-proxy" containerID="cri-o://23825b66d7ed606a30ac4c9a3825647629de142d3de7654ab9d795fc3fa46c2f" gracePeriod=30
Sep 09 20:46:08.920838 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:08.920803    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:09.220109 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220075    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5"]
Sep 09 20:46:09.220404 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220392    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d0ab72-eb5a-4318-8df3-d1ad2b62497c" containerName="cleanup"
Sep 09 20:46:09.220450 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220406    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d0ab72-eb5a-4318-8df3-d1ad2b62497c" containerName="cleanup"
Sep 09 20:46:09.220450 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220425    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef979d8-f5ac-48e7-8216-10222766c6a4" containerName="cleanup"
Sep 09 20:46:09.220450 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220431    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef979d8-f5ac-48e7-8216-10222766c6a4" containerName="cleanup"
Sep 09 20:46:09.220450 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220441    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="939047db-77d3-4306-bd4a-0449f1408193" containerName="payload-processing"
Sep 09 20:46:09.220450 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220447    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="939047db-77d3-4306-bd4a-0449f1408193" containerName="payload-processing"
Sep 09 20:46:09.220652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220458    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77198f70-b61f-47c0-a791-82efecf98205" containerName="istio-proxy"
Sep 09 20:46:09.220652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220463    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="77198f70-b61f-47c0-a791-82efecf98205" containerName="istio-proxy"
Sep 09 20:46:09.220652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220468    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeda97a1-1ac3-49bb-8ce9-a4d6f9016257" containerName="cleanup"
Sep 09 20:46:09.220652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220473    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeda97a1-1ac3-49bb-8ce9-a4d6f9016257" containerName="cleanup"
Sep 09 20:46:09.220652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220482    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77198f70-b61f-47c0-a791-82efecf98205" containerName="istio-proxy"
Sep 09 20:46:09.220652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220487    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="77198f70-b61f-47c0-a791-82efecf98205" containerName="istio-proxy"
Sep 09 20:46:09.220652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220539    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="77198f70-b61f-47c0-a791-82efecf98205" containerName="istio-proxy"
Sep 09 20:46:09.220652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220545    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeda97a1-1ac3-49bb-8ce9-a4d6f9016257" containerName="cleanup"
Sep 09 20:46:09.220652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220553    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d0ab72-eb5a-4318-8df3-d1ad2b62497c" containerName="cleanup"
Sep 09 20:46:09.220652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220559    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="939047db-77d3-4306-bd4a-0449f1408193" containerName="payload-processing"
Sep 09 20:46:09.220652 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.220564    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ef979d8-f5ac-48e7-8216-10222766c6a4" containerName="cleanup"
Sep 09 20:46:09.223179 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.223163    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5"
Sep 09 20:46:09.227446 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.227422    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:46:09.227623 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.227514    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:46:09.245136 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.245105    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5"]
Sep 09 20:46:09.301004 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.300963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8686cc7a-4fb7-4e85-8417-578e1b0e2ea0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5\" (UID: \"8686cc7a-4fb7-4e85-8417-578e1b0e2ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5"
Sep 09 20:46:09.401752 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.401705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8686cc7a-4fb7-4e85-8417-578e1b0e2ea0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5\" (UID: \"8686cc7a-4fb7-4e85-8417-578e1b0e2ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5"
Sep 09 20:46:09.402359 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.402336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8686cc7a-4fb7-4e85-8417-578e1b0e2ea0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5\" (UID: \"8686cc7a-4fb7-4e85-8417-578e1b0e2ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5"
Sep 09 20:46:09.492095 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.491996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:09.533250 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.533212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5"
Sep 09 20:46:09.816888 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:09.816862    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5"]
Sep 09 20:46:09.819490 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:46:09.819458    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8686cc7a_4fb7_4e85_8417_578e1b0e2ea0.slice/crio-e70aa50e8c6bdef8eafcb82438c7bbf48fc14a424435d4f243211a531160388d WatchSource:0}: Error finding container e70aa50e8c6bdef8eafcb82438c7bbf48fc14a424435d4f243211a531160388d: Status 404 returned error can't find the container with id e70aa50e8c6bdef8eafcb82438c7bbf48fc14a424435d4f243211a531160388d
Sep 09 20:46:10.592614 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:10.592551    2563 generic.go:356] "Generic (PLEG): container finished" podID="8686cc7a-4fb7-4e85-8417-578e1b0e2ea0" containerID="4c1e02d70a43da7de4750d87e977f772f253546f712a685f6ca662b45730615c" exitCode=0
Sep 09 20:46:10.593019 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:10.592622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5" event={"ID":"8686cc7a-4fb7-4e85-8417-578e1b0e2ea0","Type":"ContainerDied","Data":"4c1e02d70a43da7de4750d87e977f772f253546f712a685f6ca662b45730615c"}
Sep 09 20:46:10.593019 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:10.592695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5" event={"ID":"8686cc7a-4fb7-4e85-8417-578e1b0e2ea0","Type":"ContainerStarted","Data":"e70aa50e8c6bdef8eafcb82438c7bbf48fc14a424435d4f243211a531160388d"}
Sep 09 20:46:10.984973 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:10.984933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:11.740233 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:11.740206    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5"
Sep 09 20:46:11.824704 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:11.824655    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8686cc7a-4fb7-4e85-8417-578e1b0e2ea0-maas-api-service-ca\") pod \"8686cc7a-4fb7-4e85-8417-578e1b0e2ea0\" (UID: \"8686cc7a-4fb7-4e85-8417-578e1b0e2ea0\") "
Sep 09 20:46:11.825025 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:11.824997    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8686cc7a-4fb7-4e85-8417-578e1b0e2ea0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8686cc7a-4fb7-4e85-8417-578e1b0e2ea0" (UID: "8686cc7a-4fb7-4e85-8417-578e1b0e2ea0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:11.926252 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:11.926153    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8686cc7a-4fb7-4e85-8417-578e1b0e2ea0-maas-api-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:12.225461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:12.225429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:12.610594 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:12.610490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5" event={"ID":"8686cc7a-4fb7-4e85-8417-578e1b0e2ea0","Type":"ContainerDied","Data":"e70aa50e8c6bdef8eafcb82438c7bbf48fc14a424435d4f243211a531160388d"}
Sep 09 20:46:12.610594 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:12.610533    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5"
Sep 09 20:46:12.610594 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:12.610542    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e70aa50e8c6bdef8eafcb82438c7bbf48fc14a424435d4f243211a531160388d"
Sep 09 20:46:13.416280 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:13.416243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:14.093509 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.093479    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:46:14.249240 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249208    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-credential-socket\") pod \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") "
Sep 09 20:46:14.249428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249255    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnfr4\" (UniqueName: \"kubernetes.io/projected/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-kube-api-access-cnfr4\") pod \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") "
Sep 09 20:46:14.249428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249293    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-envoy\") pod \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") "
Sep 09 20:46:14.249428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249328    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istiod-ca-cert\") pod \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") "
Sep 09 20:46:14.249428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249364    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-workload-certs\") pod \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") "
Sep 09 20:46:14.249428 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249391    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-data\") pod \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") "
Sep 09 20:46:14.249710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249428    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-token\") pod \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") "
Sep 09 20:46:14.249710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249457    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-workload-socket\") pod \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") "
Sep 09 20:46:14.249710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249514    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-podinfo\") pod \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\" (UID: \"bfb7030d-c767-4bbc-9a28-171a14d5c7ec\") "
Sep 09 20:46:14.249710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249553    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bfb7030d-c767-4bbc-9a28-171a14d5c7ec" (UID: "bfb7030d-c767-4bbc-9a28-171a14d5c7ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:14.249922 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249766    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:14.249922 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249794    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-data" (OuterVolumeSpecName: "istio-data") pod "bfb7030d-c767-4bbc-9a28-171a14d5c7ec" (UID: "bfb7030d-c767-4bbc-9a28-171a14d5c7ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:14.249922 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249809    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bfb7030d-c767-4bbc-9a28-171a14d5c7ec" (UID: "bfb7030d-c767-4bbc-9a28-171a14d5c7ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:14.249922 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249828    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bfb7030d-c767-4bbc-9a28-171a14d5c7ec" (UID: "bfb7030d-c767-4bbc-9a28-171a14d5c7ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:14.249922 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.249911    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bfb7030d-c767-4bbc-9a28-171a14d5c7ec" (UID: "bfb7030d-c767-4bbc-9a28-171a14d5c7ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:14.251680 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.251624    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-kube-api-access-cnfr4" (OuterVolumeSpecName: "kube-api-access-cnfr4") pod "bfb7030d-c767-4bbc-9a28-171a14d5c7ec" (UID: "bfb7030d-c767-4bbc-9a28-171a14d5c7ec"). InnerVolumeSpecName "kube-api-access-cnfr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:14.251680 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.251647    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-token" (OuterVolumeSpecName: "istio-token") pod "bfb7030d-c767-4bbc-9a28-171a14d5c7ec" (UID: "bfb7030d-c767-4bbc-9a28-171a14d5c7ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:14.251826 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.251707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bfb7030d-c767-4bbc-9a28-171a14d5c7ec" (UID: "bfb7030d-c767-4bbc-9a28-171a14d5c7ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:46:14.251826 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.251716    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bfb7030d-c767-4bbc-9a28-171a14d5c7ec" (UID: "bfb7030d-c767-4bbc-9a28-171a14d5c7ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:14.350951 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.350910    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:14.350951 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.350943    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:14.350951 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.350954    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:14.350951 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.350962    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:14.351215 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.350973    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:14.351215 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.350984    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnfr4\" (UniqueName: \"kubernetes.io/projected/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-kube-api-access-cnfr4\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:14.351215 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.350993    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:14.351215 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.351001    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb7030d-c767-4bbc-9a28-171a14d5c7ec-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:14.415040 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.414998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:14.621333 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.621243    2563 generic.go:356] "Generic (PLEG): container finished" podID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerID="23825b66d7ed606a30ac4c9a3825647629de142d3de7654ab9d795fc3fa46c2f" exitCode=0
Sep 09 20:46:14.621333 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.621298    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"
Sep 09 20:46:14.621857 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.621333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" event={"ID":"bfb7030d-c767-4bbc-9a28-171a14d5c7ec","Type":"ContainerDied","Data":"23825b66d7ed606a30ac4c9a3825647629de142d3de7654ab9d795fc3fa46c2f"}
Sep 09 20:46:14.621857 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.621389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w" event={"ID":"bfb7030d-c767-4bbc-9a28-171a14d5c7ec","Type":"ContainerDied","Data":"ea8f3f0329b87ffa11e821bfa389e7e993692539ebeeb1b076032a32357ca2f9"}
Sep 09 20:46:14.621857 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.621412    2563 scope.go:117] "RemoveContainer" containerID="23825b66d7ed606a30ac4c9a3825647629de142d3de7654ab9d795fc3fa46c2f"
Sep 09 20:46:14.630889 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.630869    2563 scope.go:117] "RemoveContainer" containerID="ff115c0b2a665d65eb061018bc8c47b498e40ccaaf26ab8a35d6ceee810362f2"
Sep 09 20:46:14.638844 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.638827    2563 scope.go:117] "RemoveContainer" containerID="23825b66d7ed606a30ac4c9a3825647629de142d3de7654ab9d795fc3fa46c2f"
Sep 09 20:46:14.639111 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:46:14.639092    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23825b66d7ed606a30ac4c9a3825647629de142d3de7654ab9d795fc3fa46c2f\": container with ID starting with 23825b66d7ed606a30ac4c9a3825647629de142d3de7654ab9d795fc3fa46c2f not found: ID does not exist" containerID="23825b66d7ed606a30ac4c9a3825647629de142d3de7654ab9d795fc3fa46c2f"
Sep 09 20:46:14.639182 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.639119    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23825b66d7ed606a30ac4c9a3825647629de142d3de7654ab9d795fc3fa46c2f"} err="failed to get container status \"23825b66d7ed606a30ac4c9a3825647629de142d3de7654ab9d795fc3fa46c2f\": rpc error: code = NotFound desc = could not find container \"23825b66d7ed606a30ac4c9a3825647629de142d3de7654ab9d795fc3fa46c2f\": container with ID starting with 23825b66d7ed606a30ac4c9a3825647629de142d3de7654ab9d795fc3fa46c2f not found: ID does not exist"
Sep 09 20:46:14.639182 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.639136    2563 scope.go:117] "RemoveContainer" containerID="ff115c0b2a665d65eb061018bc8c47b498e40ccaaf26ab8a35d6ceee810362f2"
Sep 09 20:46:14.639447 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:46:14.639431    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff115c0b2a665d65eb061018bc8c47b498e40ccaaf26ab8a35d6ceee810362f2\": container with ID starting with ff115c0b2a665d65eb061018bc8c47b498e40ccaaf26ab8a35d6ceee810362f2 not found: ID does not exist" containerID="ff115c0b2a665d65eb061018bc8c47b498e40ccaaf26ab8a35d6ceee810362f2"
Sep 09 20:46:14.639490 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.639451    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff115c0b2a665d65eb061018bc8c47b498e40ccaaf26ab8a35d6ceee810362f2"} err="failed to get container status \"ff115c0b2a665d65eb061018bc8c47b498e40ccaaf26ab8a35d6ceee810362f2\": rpc error: code = NotFound desc = could not find container \"ff115c0b2a665d65eb061018bc8c47b498e40ccaaf26ab8a35d6ceee810362f2\": container with ID starting with ff115c0b2a665d65eb061018bc8c47b498e40ccaaf26ab8a35d6ceee810362f2 not found: ID does not exist"
Sep 09 20:46:14.646355 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.646331    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"]
Sep 09 20:46:14.652814 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:14.652791    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-868e40-openshift-default-794b74b896-7lr4w"]
Sep 09 20:46:15.315421 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:15.315386    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" path="/var/lib/kubelet/pods/bfb7030d-c767-4bbc-9a28-171a14d5c7ec/volumes"
Sep 09 20:46:20.210487 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:20.210448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:20.387249 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:20.387208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:20.540115 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:20.539998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:20.661317 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:20.661286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:20.694761 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:20.694715    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5"]
Sep 09 20:46:20.701541 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:20.701508    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-697f19-9txj5"]
Sep 09 20:46:20.797042 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:20.796946    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:21.164802 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:21.164697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:21.313027 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:21.312992    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8686cc7a-4fb7-4e85-8417-578e1b0e2ea0" path="/var/lib/kubelet/pods/8686cc7a-4fb7-4e85-8417-578e1b0e2ea0/volumes"
Sep 09 20:46:21.384819 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:21.384781    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:23.012144 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:23.012105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:23.659946 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:23.659916    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:46:24.148816 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:24.148778    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:28.409769 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:28.409682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:28.573684 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:28.573645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:28.899657 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:28.899623    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"]
Sep 09 20:46:28.899863 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:28.899830    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy" containerID="cri-o://89916c5b2442a52d7be518ce3d1b446cfacc31c9b64eca11d25ff62470a007d1" gracePeriod=30
Sep 09 20:46:28.928934 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:28.928904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:29.251188 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.251155    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l"]
Sep 09 20:46:29.251512 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.251497    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8686cc7a-4fb7-4e85-8417-578e1b0e2ea0" containerName="revoke-keys"
Sep 09 20:46:29.251558 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.251516    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8686cc7a-4fb7-4e85-8417-578e1b0e2ea0" containerName="revoke-keys"
Sep 09 20:46:29.251558 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.251541    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerName="istio-proxy"
Sep 09 20:46:29.251558 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.251547    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerName="istio-proxy"
Sep 09 20:46:29.251558 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.251553    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerName="istio-proxy"
Sep 09 20:46:29.251558 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.251558    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerName="istio-proxy"
Sep 09 20:46:29.251728 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.251624    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerName="istio-proxy"
Sep 09 20:46:29.251728 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.251635    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="77198f70-b61f-47c0-a791-82efecf98205" containerName="istio-proxy"
Sep 09 20:46:29.251728 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.251642    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8686cc7a-4fb7-4e85-8417-578e1b0e2ea0" containerName="revoke-keys"
Sep 09 20:46:29.255955 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.255929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l"
Sep 09 20:46:29.260115 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.260083    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:46:29.260247 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.260137    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:46:29.283125 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.283091    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l"]
Sep 09 20:46:29.382657 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.382618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a27f90-7a2b-4964-ac47-178383b92233-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l\" (UID: \"01a27f90-7a2b-4964-ac47-178383b92233\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l"
Sep 09 20:46:29.483421 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.483381    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a27f90-7a2b-4964-ac47-178383b92233-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l\" (UID: \"01a27f90-7a2b-4964-ac47-178383b92233\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l"
Sep 09 20:46:29.484027 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.484007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a27f90-7a2b-4964-ac47-178383b92233-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l\" (UID: \"01a27f90-7a2b-4964-ac47-178383b92233\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l"
Sep 09 20:46:29.565988 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.565897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l"
Sep 09 20:46:29.844506 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:29.844450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l"]
Sep 09 20:46:29.847351 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:46:29.847324    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a27f90_7a2b_4964_ac47_178383b92233.slice/crio-59acd57ff18199eb0251ace385a2375ac80c6611742980f3cfba4a5f596b2e3d WatchSource:0}: Error finding container 59acd57ff18199eb0251ace385a2375ac80c6611742980f3cfba4a5f596b2e3d: Status 404 returned error can't find the container with id 59acd57ff18199eb0251ace385a2375ac80c6611742980f3cfba4a5f596b2e3d
Sep 09 20:46:30.389005 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:30.388966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:30.696758 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:30.696725    2563 generic.go:356] "Generic (PLEG): container finished" podID="01a27f90-7a2b-4964-ac47-178383b92233" containerID="3775ddcb4616faef67ae499309d2dcb980065d97ec19f1e7352bbc7493a87bf2" exitCode=0
Sep 09 20:46:30.697092 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:30.696805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l" event={"ID":"01a27f90-7a2b-4964-ac47-178383b92233","Type":"ContainerDied","Data":"3775ddcb4616faef67ae499309d2dcb980065d97ec19f1e7352bbc7493a87bf2"}
Sep 09 20:46:30.697092 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:30.696859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l" event={"ID":"01a27f90-7a2b-4964-ac47-178383b92233","Type":"ContainerStarted","Data":"59acd57ff18199eb0251ace385a2375ac80c6611742980f3cfba4a5f596b2e3d"}
Sep 09 20:46:31.740180 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:31.740089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:31.833892 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:31.833866    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l"
Sep 09 20:46:32.007344 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:32.007251    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a27f90-7a2b-4964-ac47-178383b92233-maas-api-service-ca\") pod \"01a27f90-7a2b-4964-ac47-178383b92233\" (UID: \"01a27f90-7a2b-4964-ac47-178383b92233\") "
Sep 09 20:46:32.007589 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:32.007551    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a27f90-7a2b-4964-ac47-178383b92233-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "01a27f90-7a2b-4964-ac47-178383b92233" (UID: "01a27f90-7a2b-4964-ac47-178383b92233"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:32.107877 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:32.107837    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a27f90-7a2b-4964-ac47-178383b92233-maas-api-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:32.707649 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:32.707613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l" event={"ID":"01a27f90-7a2b-4964-ac47-178383b92233","Type":"ContainerDied","Data":"59acd57ff18199eb0251ace385a2375ac80c6611742980f3cfba4a5f596b2e3d"}
Sep 09 20:46:32.707649 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:32.707643    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l"
Sep 09 20:46:32.707869 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:32.707663    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59acd57ff18199eb0251ace385a2375ac80c6611742980f3cfba4a5f596b2e3d"
Sep 09 20:46:32.966203 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:32.966108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:34.158648 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.158620    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:46:34.225300 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.225268    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-istio-data\") pod \"f0732bd8-480d-442f-be41-f4363ad21c35\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") "
Sep 09 20:46:34.225461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.225324    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-workload-socket\") pod \"f0732bd8-480d-442f-be41-f4363ad21c35\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") "
Sep 09 20:46:34.225461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.225378    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0732bd8-480d-442f-be41-f4363ad21c35-istio-podinfo\") pod \"f0732bd8-480d-442f-be41-f4363ad21c35\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") "
Sep 09 20:46:34.225461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.225419    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-workload-certs\") pod \"f0732bd8-480d-442f-be41-f4363ad21c35\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") "
Sep 09 20:46:34.225461 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.225443    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0732bd8-480d-442f-be41-f4363ad21c35-istio-token\") pod \"f0732bd8-480d-442f-be41-f4363ad21c35\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") "
Sep 09 20:46:34.225659 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.225475    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-istio-envoy\") pod \"f0732bd8-480d-442f-be41-f4363ad21c35\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") "
Sep 09 20:46:34.225659 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.225488    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0732bd8-480d-442f-be41-f4363ad21c35" (UID: "f0732bd8-480d-442f-be41-f4363ad21c35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:34.225659 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.225499    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0732bd8-480d-442f-be41-f4363ad21c35-istiod-ca-cert\") pod \"f0732bd8-480d-442f-be41-f4363ad21c35\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") "
Sep 09 20:46:34.225659 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.225620    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0732bd8-480d-442f-be41-f4363ad21c35" (UID: "f0732bd8-480d-442f-be41-f4363ad21c35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:34.225659 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.225647    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-credential-socket\") pod \"f0732bd8-480d-442f-be41-f4363ad21c35\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") "
Sep 09 20:46:34.225906 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.225695    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtjdg\" (UniqueName: \"kubernetes.io/projected/f0732bd8-480d-442f-be41-f4363ad21c35-kube-api-access-vtjdg\") pod \"f0732bd8-480d-442f-be41-f4363ad21c35\" (UID: \"f0732bd8-480d-442f-be41-f4363ad21c35\") "
Sep 09 20:46:34.225955 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.225900    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0732bd8-480d-442f-be41-f4363ad21c35" (UID: "f0732bd8-480d-442f-be41-f4363ad21c35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:34.225955 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.225928    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0732bd8-480d-442f-be41-f4363ad21c35-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0732bd8-480d-442f-be41-f4363ad21c35" (UID: "f0732bd8-480d-442f-be41-f4363ad21c35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:34.226071 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.226015    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0732bd8-480d-442f-be41-f4363ad21c35-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:34.226125 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.226097    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0732bd8-480d-442f-be41-f4363ad21c35" (UID: "f0732bd8-480d-442f-be41-f4363ad21c35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:34.226192 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.226131    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:34.226192 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.226149    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:34.226192 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.226163    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:34.227786 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.227754    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0732bd8-480d-442f-be41-f4363ad21c35" (UID: "f0732bd8-480d-442f-be41-f4363ad21c35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:34.227919 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.227813    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0732bd8-480d-442f-be41-f4363ad21c35-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0732bd8-480d-442f-be41-f4363ad21c35" (UID: "f0732bd8-480d-442f-be41-f4363ad21c35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:46:34.227919 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.227812    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0732bd8-480d-442f-be41-f4363ad21c35-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0732bd8-480d-442f-be41-f4363ad21c35" (UID: "f0732bd8-480d-442f-be41-f4363ad21c35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:34.227919 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.227884    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0732bd8-480d-442f-be41-f4363ad21c35-kube-api-access-vtjdg" (OuterVolumeSpecName: "kube-api-access-vtjdg") pod "f0732bd8-480d-442f-be41-f4363ad21c35" (UID: "f0732bd8-480d-442f-be41-f4363ad21c35"). InnerVolumeSpecName "kube-api-access-vtjdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:34.327343 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.327297    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtjdg\" (UniqueName: \"kubernetes.io/projected/f0732bd8-480d-442f-be41-f4363ad21c35-kube-api-access-vtjdg\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:34.327343 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.327335    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0732bd8-480d-442f-be41-f4363ad21c35-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:34.327343 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.327348    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:34.327620 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.327358    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0732bd8-480d-442f-be41-f4363ad21c35-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:34.327620 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.327370    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0732bd8-480d-442f-be41-f4363ad21c35-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:34.718750 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.718718    2563 generic.go:356] "Generic (PLEG): container finished" podID="f0732bd8-480d-442f-be41-f4363ad21c35" containerID="89916c5b2442a52d7be518ce3d1b446cfacc31c9b64eca11d25ff62470a007d1" exitCode=0
Sep 09 20:46:34.718965 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.718774    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"
Sep 09 20:46:34.718965 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.718802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" event={"ID":"f0732bd8-480d-442f-be41-f4363ad21c35","Type":"ContainerDied","Data":"89916c5b2442a52d7be518ce3d1b446cfacc31c9b64eca11d25ff62470a007d1"}
Sep 09 20:46:34.718965 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.718856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf" event={"ID":"f0732bd8-480d-442f-be41-f4363ad21c35","Type":"ContainerDied","Data":"71f0374c3fdbc8ce1901c7c9e740f9f7d7db415d6c5b0683cb3bc18bf88dccf6"}
Sep 09 20:46:34.718965 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.718878    2563 scope.go:117] "RemoveContainer" containerID="89916c5b2442a52d7be518ce3d1b446cfacc31c9b64eca11d25ff62470a007d1"
Sep 09 20:46:34.728094 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.728071    2563 scope.go:117] "RemoveContainer" containerID="d109952bce8076364e3a10526f3a0ad055a91ef1722991546c7c5bef10b00a94"
Sep 09 20:46:34.736286 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.736266    2563 scope.go:117] "RemoveContainer" containerID="89916c5b2442a52d7be518ce3d1b446cfacc31c9b64eca11d25ff62470a007d1"
Sep 09 20:46:34.736619 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:46:34.736593    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89916c5b2442a52d7be518ce3d1b446cfacc31c9b64eca11d25ff62470a007d1\": container with ID starting with 89916c5b2442a52d7be518ce3d1b446cfacc31c9b64eca11d25ff62470a007d1 not found: ID does not exist" containerID="89916c5b2442a52d7be518ce3d1b446cfacc31c9b64eca11d25ff62470a007d1"
Sep 09 20:46:34.736734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.736625    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89916c5b2442a52d7be518ce3d1b446cfacc31c9b64eca11d25ff62470a007d1"} err="failed to get container status \"89916c5b2442a52d7be518ce3d1b446cfacc31c9b64eca11d25ff62470a007d1\": rpc error: code = NotFound desc = could not find container \"89916c5b2442a52d7be518ce3d1b446cfacc31c9b64eca11d25ff62470a007d1\": container with ID starting with 89916c5b2442a52d7be518ce3d1b446cfacc31c9b64eca11d25ff62470a007d1 not found: ID does not exist"
Sep 09 20:46:34.736734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.736646    2563 scope.go:117] "RemoveContainer" containerID="d109952bce8076364e3a10526f3a0ad055a91ef1722991546c7c5bef10b00a94"
Sep 09 20:46:34.736949 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:46:34.736930    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d109952bce8076364e3a10526f3a0ad055a91ef1722991546c7c5bef10b00a94\": container with ID starting with d109952bce8076364e3a10526f3a0ad055a91ef1722991546c7c5bef10b00a94 not found: ID does not exist" containerID="d109952bce8076364e3a10526f3a0ad055a91ef1722991546c7c5bef10b00a94"
Sep 09 20:46:34.737000 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.736955    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d109952bce8076364e3a10526f3a0ad055a91ef1722991546c7c5bef10b00a94"} err="failed to get container status \"d109952bce8076364e3a10526f3a0ad055a91ef1722991546c7c5bef10b00a94\": rpc error: code = NotFound desc = could not find container \"d109952bce8076364e3a10526f3a0ad055a91ef1722991546c7c5bef10b00a94\": container with ID starting with d109952bce8076364e3a10526f3a0ad055a91ef1722991546c7c5bef10b00a94 not found: ID does not exist"
Sep 09 20:46:34.747321 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.747297    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"]
Sep 09 20:46:34.752314 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:34.752292    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-697f19-openshift-default-5cdcd594fb-9vfcf"]
Sep 09 20:46:35.313162 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:35.313117    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" path="/var/lib/kubelet/pods/f0732bd8-480d-442f-be41-f4363ad21c35/volumes"
Sep 09 20:46:40.170038 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.170001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:40.232772 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.232734    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"]
Sep 09 20:46:40.233040 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.233002    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h" podUID="8dbf3e43-5a97-4dd7-8301-2ac711f8d08c" containerName="payload-processing" containerID="cri-o://0f5803536a397339ce4ef55a9966efc31c7904611ad6e893a16e421b4c8ce014" gracePeriod=30
Sep 09 20:46:40.308777 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.308743    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:40.432749 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.432656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:40.493090 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.493064    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"
Sep 09 20:46:40.579277 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.579229    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ht65\" (UniqueName: \"kubernetes.io/projected/8dbf3e43-5a97-4dd7-8301-2ac711f8d08c-kube-api-access-2ht65\") pod \"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c\" (UID: \"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c\") "
Sep 09 20:46:40.579474 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.579338    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbf3e43-5a97-4dd7-8301-2ac711f8d08c-plugins-config-volume\") pod \"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c\" (UID: \"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c\") "
Sep 09 20:46:40.579750 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.579712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbf3e43-5a97-4dd7-8301-2ac711f8d08c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8dbf3e43-5a97-4dd7-8301-2ac711f8d08c" (UID: "8dbf3e43-5a97-4dd7-8301-2ac711f8d08c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:40.580160 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.580126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:40.581526 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.581501    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbf3e43-5a97-4dd7-8301-2ac711f8d08c-kube-api-access-2ht65" (OuterVolumeSpecName: "kube-api-access-2ht65") pod "8dbf3e43-5a97-4dd7-8301-2ac711f8d08c" (UID: "8dbf3e43-5a97-4dd7-8301-2ac711f8d08c"). InnerVolumeSpecName "kube-api-access-2ht65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:40.671917 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.671871    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l"]
Sep 09 20:46:40.680340 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.680312    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dbf3e43-5a97-4dd7-8301-2ac711f8d08c-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:40.680340 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.680337    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ht65\" (UniqueName: \"kubernetes.io/projected/8dbf3e43-5a97-4dd7-8301-2ac711f8d08c-kube-api-access-2ht65\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:40.681830 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.681807    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-166b94-n497l"]
Sep 09 20:46:40.736019 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.735988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:40.748157 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.748132    2563 generic.go:356] "Generic (PLEG): container finished" podID="8dbf3e43-5a97-4dd7-8301-2ac711f8d08c" containerID="0f5803536a397339ce4ef55a9966efc31c7904611ad6e893a16e421b4c8ce014" exitCode=0
Sep 09 20:46:40.748307 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.748182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h" event={"ID":"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c","Type":"ContainerDied","Data":"0f5803536a397339ce4ef55a9966efc31c7904611ad6e893a16e421b4c8ce014"}
Sep 09 20:46:40.748307 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.748186    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"
Sep 09 20:46:40.748307 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.748225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h" event={"ID":"8dbf3e43-5a97-4dd7-8301-2ac711f8d08c","Type":"ContainerDied","Data":"f53177113d2a59c8a721fbd4633a75edf973eec4acd1ff7a9a57681fbfec8b67"}
Sep 09 20:46:40.748307 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.748247    2563 scope.go:117] "RemoveContainer" containerID="0f5803536a397339ce4ef55a9966efc31c7904611ad6e893a16e421b4c8ce014"
Sep 09 20:46:40.758063 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.757702    2563 scope.go:117] "RemoveContainer" containerID="0f5803536a397339ce4ef55a9966efc31c7904611ad6e893a16e421b4c8ce014"
Sep 09 20:46:40.760198 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:46:40.758361    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5803536a397339ce4ef55a9966efc31c7904611ad6e893a16e421b4c8ce014\": container with ID starting with 0f5803536a397339ce4ef55a9966efc31c7904611ad6e893a16e421b4c8ce014 not found: ID does not exist" containerID="0f5803536a397339ce4ef55a9966efc31c7904611ad6e893a16e421b4c8ce014"
Sep 09 20:46:40.760198 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.758403    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5803536a397339ce4ef55a9966efc31c7904611ad6e893a16e421b4c8ce014"} err="failed to get container status \"0f5803536a397339ce4ef55a9966efc31c7904611ad6e893a16e421b4c8ce014\": rpc error: code = NotFound desc = could not find container \"0f5803536a397339ce4ef55a9966efc31c7904611ad6e893a16e421b4c8ce014\": container with ID starting with 0f5803536a397339ce4ef55a9966efc31c7904611ad6e893a16e421b4c8ce014 not found: ID does not exist"
Sep 09 20:46:40.794920 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.794884    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"]
Sep 09 20:46:40.807436 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:40.807406    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-166b94-6f87f9bfff-7c97h"]
Sep 09 20:46:41.093349 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:41.093255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:41.293940 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:41.293906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:41.313448 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:41.313420    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01a27f90-7a2b-4964-ac47-178383b92233" path="/var/lib/kubelet/pods/01a27f90-7a2b-4964-ac47-178383b92233/volumes"
Sep 09 20:46:41.313862 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:41.313846    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dbf3e43-5a97-4dd7-8301-2ac711f8d08c" path="/var/lib/kubelet/pods/8dbf3e43-5a97-4dd7-8301-2ac711f8d08c/volumes"
Sep 09 20:46:42.361734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:42.361690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:43.365023 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:43.364987    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:46:49.291105 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:49.291071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:49.556761 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:49.556668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:49.800926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:49.800883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:50.185541 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.185481    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9"]
Sep 09 20:46:50.185908 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.185890    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dbf3e43-5a97-4dd7-8301-2ac711f8d08c" containerName="payload-processing"
Sep 09 20:46:50.185908 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.185909    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dbf3e43-5a97-4dd7-8301-2ac711f8d08c" containerName="payload-processing"
Sep 09 20:46:50.186009 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.185919    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a27f90-7a2b-4964-ac47-178383b92233" containerName="revoke-keys"
Sep 09 20:46:50.186009 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.185925    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a27f90-7a2b-4964-ac47-178383b92233" containerName="revoke-keys"
Sep 09 20:46:50.186009 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.185938    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy"
Sep 09 20:46:50.186009 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.185944    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy"
Sep 09 20:46:50.186009 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.185950    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy"
Sep 09 20:46:50.186009 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.185955    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy"
Sep 09 20:46:50.186209 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.186014    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy"
Sep 09 20:46:50.186209 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.186024    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dbf3e43-5a97-4dd7-8301-2ac711f8d08c" containerName="payload-processing"
Sep 09 20:46:50.186209 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.186030    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb7030d-c767-4bbc-9a28-171a14d5c7ec" containerName="istio-proxy"
Sep 09 20:46:50.186209 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.186037    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01a27f90-7a2b-4964-ac47-178383b92233" containerName="revoke-keys"
Sep 09 20:46:50.186209 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.186043    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0732bd8-480d-442f-be41-f4363ad21c35" containerName="istio-proxy"
Sep 09 20:46:50.189273 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.189252    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9"
Sep 09 20:46:50.194438 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.194413    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:46:50.196209 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.196186    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:46:50.236452 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.236418    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9"]
Sep 09 20:46:50.261422 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.261385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72218f80-ffb8-488d-9062-546bd599523f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9\" (UID: \"72218f80-ffb8-488d-9062-546bd599523f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9"
Sep 09 20:46:50.361984 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.361941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72218f80-ffb8-488d-9062-546bd599523f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9\" (UID: \"72218f80-ffb8-488d-9062-546bd599523f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9"
Sep 09 20:46:50.362602 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.362556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72218f80-ffb8-488d-9062-546bd599523f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9\" (UID: \"72218f80-ffb8-488d-9062-546bd599523f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9"
Sep 09 20:46:50.499299 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.499258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9"
Sep 09 20:46:50.554095 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.553942    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:50.822773 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:50.822694    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9"]
Sep 09 20:46:50.826557 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:46:50.826528    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72218f80_ffb8_488d_9062_546bd599523f.slice/crio-2a2b5b7ee4f36078822d1555dd3511231f2e964518dca5f6d70fb72693925e21 WatchSource:0}: Error finding container 2a2b5b7ee4f36078822d1555dd3511231f2e964518dca5f6d70fb72693925e21: Status 404 returned error can't find the container with id 2a2b5b7ee4f36078822d1555dd3511231f2e964518dca5f6d70fb72693925e21
Sep 09 20:46:51.801552 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:51.801465    2563 generic.go:356] "Generic (PLEG): container finished" podID="72218f80-ffb8-488d-9062-546bd599523f" containerID="6f2a64c024594cf74960a6e4b68708c17c15394b50fdeb54e810169ff5231b75" exitCode=0
Sep 09 20:46:51.801952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:51.801555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9" event={"ID":"72218f80-ffb8-488d-9062-546bd599523f","Type":"ContainerDied","Data":"6f2a64c024594cf74960a6e4b68708c17c15394b50fdeb54e810169ff5231b75"}
Sep 09 20:46:51.801952 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:51.801671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9" event={"ID":"72218f80-ffb8-488d-9062-546bd599523f","Type":"ContainerStarted","Data":"2a2b5b7ee4f36078822d1555dd3511231f2e964518dca5f6d70fb72693925e21"}
Sep 09 20:46:52.676174 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:52.676133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:52.929858 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:52.929796    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9"
Sep 09 20:46:52.985587 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:52.985549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72218f80-ffb8-488d-9062-546bd599523f-maas-api-service-ca\") pod \"72218f80-ffb8-488d-9062-546bd599523f\" (UID: \"72218f80-ffb8-488d-9062-546bd599523f\") "
Sep 09 20:46:52.985923 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:52.985899    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72218f80-ffb8-488d-9062-546bd599523f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72218f80-ffb8-488d-9062-546bd599523f" (UID: "72218f80-ffb8-488d-9062-546bd599523f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:53.087112 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:53.087078    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72218f80-ffb8-488d-9062-546bd599523f-maas-api-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:46:53.812479 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:53.812448    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9"
Sep 09 20:46:53.812678 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:53.812484    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9" event={"ID":"72218f80-ffb8-488d-9062-546bd599523f","Type":"ContainerDied","Data":"2a2b5b7ee4f36078822d1555dd3511231f2e964518dca5f6d70fb72693925e21"}
Sep 09 20:46:53.812678 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:53.812531    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a2b5b7ee4f36078822d1555dd3511231f2e964518dca5f6d70fb72693925e21"
Sep 09 20:46:54.553374 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:46:54.553339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:01.120923 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:01.120886    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d"]
Sep 09 20:47:01.125734 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:01.125708    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4101f2-29816445-mwm9d"]
Sep 09 20:47:01.150094 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:01.150054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:01.313537 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:01.313503    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d0ab72-eb5a-4318-8df3-d1ad2b62497c" path="/var/lib/kubelet/pods/06d0ab72-eb5a-4318-8df3-d1ad2b62497c/volumes"
Sep 09 20:47:01.411276 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:01.411198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:01.774541 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:01.774500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:01.817125 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:01.817086    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9"]
Sep 09 20:47:01.822240 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:01.822208    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4101f2-k4vq9"]
Sep 09 20:47:01.983710 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:01.983676    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:02.432799 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:02.432763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:02.930960 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:02.930921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:03.312824 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:03.312791    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72218f80-ffb8-488d-9062-546bd599523f" path="/var/lib/kubelet/pods/72218f80-ffb8-488d-9062-546bd599523f/volumes"
Sep 09 20:47:04.181977 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:04.181932    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:47:10.758726 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:10.758685    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:11.084348 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:11.084265    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"]
Sep 09 20:47:11.084563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:11.084536    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerName="istio-proxy" containerID="cri-o://b1d93ca98ac60866727a2eb68794c9d1e96597186dfbf8edec0ef2b9537643ab" gracePeriod=30
Sep 09 20:47:12.643897 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:12.643802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:15.551112 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:15.551074    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:47:15.805532 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:15.805438    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 20:47:15.805532 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:15.805497    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:16.326448 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.326422    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:47:16.489347 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.489310    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istiod-ca-cert\") pod \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") "
Sep 09 20:47:16.489546 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.489378    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-workload-socket\") pod \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") "
Sep 09 20:47:16.489546 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.489394    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-data\") pod \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") "
Sep 09 20:47:16.489546 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.489425    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xqcv\" (UniqueName: \"kubernetes.io/projected/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-kube-api-access-6xqcv\") pod \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") "
Sep 09 20:47:16.489756 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.489623    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-credential-socket\") pod \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") "
Sep 09 20:47:16.489756 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.489676    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-envoy\") pod \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") "
Sep 09 20:47:16.489756 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.489724    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-podinfo\") pod \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") "
Sep 09 20:47:16.489756 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.489743    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" (UID: "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:16.489962 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.489755    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-data" (OuterVolumeSpecName: "istio-data") pod "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" (UID: "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:16.489962 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.489766    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-token\") pod \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") "
Sep 09 20:47:16.489962 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.489793    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-workload-certs\") pod \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\" (UID: \"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe\") "
Sep 09 20:47:16.489962 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.489812    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" (UID: "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:47:16.489962 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.489924    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" (UID: "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:16.490180 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.490115    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:47:16.490180 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.490135    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:47:16.490180 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.490149    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:47:16.490180 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.490162    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:47:16.490397 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.490349    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" (UID: "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:16.491722 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.491689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-kube-api-access-6xqcv" (OuterVolumeSpecName: "kube-api-access-6xqcv") pod "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" (UID: "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe"). InnerVolumeSpecName "kube-api-access-6xqcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:16.491844 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.491790    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-token" (OuterVolumeSpecName: "istio-token") pod "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" (UID: "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:16.491893 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.491847    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" (UID: "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:47:16.491893 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.491876    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" (UID: "edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:16.591149 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.591113    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xqcv\" (UniqueName: \"kubernetes.io/projected/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-kube-api-access-6xqcv\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:47:16.591149 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.591143    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:47:16.591149 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.591156    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:47:16.591605 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.591165    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:47:16.591605 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.591173    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:47:16.922848 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.922756    2563 generic.go:356] "Generic (PLEG): container finished" podID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerID="b1d93ca98ac60866727a2eb68794c9d1e96597186dfbf8edec0ef2b9537643ab" exitCode=0
Sep 09 20:47:16.922848 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.922817    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"
Sep 09 20:47:16.923081 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.922842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" event={"ID":"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe","Type":"ContainerDied","Data":"b1d93ca98ac60866727a2eb68794c9d1e96597186dfbf8edec0ef2b9537643ab"}
Sep 09 20:47:16.923081 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.922905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb" event={"ID":"edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe","Type":"ContainerDied","Data":"259c70d5c55428820d232a44899171ab6959254d9ecfbf391d7295d051117b3d"}
Sep 09 20:47:16.923081 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.922935    2563 scope.go:117] "RemoveContainer" containerID="b1d93ca98ac60866727a2eb68794c9d1e96597186dfbf8edec0ef2b9537643ab"
Sep 09 20:47:16.932029 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.932012    2563 scope.go:117] "RemoveContainer" containerID="ea030676ed85f612b4350c1796f1eda6b6b1a59bac098ef6852b60c1e1744e0c"
Sep 09 20:47:16.940013 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.939993    2563 scope.go:117] "RemoveContainer" containerID="b1d93ca98ac60866727a2eb68794c9d1e96597186dfbf8edec0ef2b9537643ab"
Sep 09 20:47:16.940301 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:47:16.940280    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d93ca98ac60866727a2eb68794c9d1e96597186dfbf8edec0ef2b9537643ab\": container with ID starting with b1d93ca98ac60866727a2eb68794c9d1e96597186dfbf8edec0ef2b9537643ab not found: ID does not exist" containerID="b1d93ca98ac60866727a2eb68794c9d1e96597186dfbf8edec0ef2b9537643ab"
Sep 09 20:47:16.940357 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.940309    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d93ca98ac60866727a2eb68794c9d1e96597186dfbf8edec0ef2b9537643ab"} err="failed to get container status \"b1d93ca98ac60866727a2eb68794c9d1e96597186dfbf8edec0ef2b9537643ab\": rpc error: code = NotFound desc = could not find container \"b1d93ca98ac60866727a2eb68794c9d1e96597186dfbf8edec0ef2b9537643ab\": container with ID starting with b1d93ca98ac60866727a2eb68794c9d1e96597186dfbf8edec0ef2b9537643ab not found: ID does not exist"
Sep 09 20:47:16.940357 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.940329    2563 scope.go:117] "RemoveContainer" containerID="ea030676ed85f612b4350c1796f1eda6b6b1a59bac098ef6852b60c1e1744e0c"
Sep 09 20:47:16.940603 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:47:16.940567    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea030676ed85f612b4350c1796f1eda6b6b1a59bac098ef6852b60c1e1744e0c\": container with ID starting with ea030676ed85f612b4350c1796f1eda6b6b1a59bac098ef6852b60c1e1744e0c not found: ID does not exist" containerID="ea030676ed85f612b4350c1796f1eda6b6b1a59bac098ef6852b60c1e1744e0c"
Sep 09 20:47:16.940649 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.940607    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea030676ed85f612b4350c1796f1eda6b6b1a59bac098ef6852b60c1e1744e0c"} err="failed to get container status \"ea030676ed85f612b4350c1796f1eda6b6b1a59bac098ef6852b60c1e1744e0c\": rpc error: code = NotFound desc = could not find container \"ea030676ed85f612b4350c1796f1eda6b6b1a59bac098ef6852b60c1e1744e0c\": container with ID starting with ea030676ed85f612b4350c1796f1eda6b6b1a59bac098ef6852b60c1e1744e0c not found: ID does not exist"
Sep 09 20:47:16.952683 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.952649    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"]
Sep 09 20:47:16.958352 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:16.958326    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4101f2-openshift-default-6d456cc6c4-mlfdb"]
Sep 09 20:47:17.313488 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:47:17.313447    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" path="/var/lib/kubelet/pods/edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe/volumes"
Sep 09 20:48:39.524812 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:48:39.524772    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:48:40.115189 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:48:40.115149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:41.551958 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:48:41.551914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:04.617567 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:04.617535    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:49:13.007236 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:13.007200    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:49:20.164047 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:20.164001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:20.827010 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:20.826971    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:49:34.810178 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:34.810145    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:49:43.099043 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:43.099006    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:49:49.901501 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:49.901462    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"]
Sep 09 20:49:49.901919 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:49.901707    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj" podUID="d8ef4400-3124-45b5-8a46-402b7b38f6e4" containerName="manager" containerID="cri-o://fdfea039764fd0c2ea55cc6f275268877e85bddf92e03bb606bc2a21bb85d851" gracePeriod=10
Sep 09 20:49:50.157384 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.157316    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"
Sep 09 20:49:50.273058 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.273020    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8ef4400-3124-45b5-8a46-402b7b38f6e4-extensions-socket-volume\") pod \"d8ef4400-3124-45b5-8a46-402b7b38f6e4\" (UID: \"d8ef4400-3124-45b5-8a46-402b7b38f6e4\") "
Sep 09 20:49:50.273232 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.273121    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsr7q\" (UniqueName: \"kubernetes.io/projected/d8ef4400-3124-45b5-8a46-402b7b38f6e4-kube-api-access-vsr7q\") pod \"d8ef4400-3124-45b5-8a46-402b7b38f6e4\" (UID: \"d8ef4400-3124-45b5-8a46-402b7b38f6e4\") "
Sep 09 20:49:50.273407 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.273384    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ef4400-3124-45b5-8a46-402b7b38f6e4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d8ef4400-3124-45b5-8a46-402b7b38f6e4" (UID: "d8ef4400-3124-45b5-8a46-402b7b38f6e4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:50.275146 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.275125    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ef4400-3124-45b5-8a46-402b7b38f6e4-kube-api-access-vsr7q" (OuterVolumeSpecName: "kube-api-access-vsr7q") pod "d8ef4400-3124-45b5-8a46-402b7b38f6e4" (UID: "d8ef4400-3124-45b5-8a46-402b7b38f6e4"). InnerVolumeSpecName "kube-api-access-vsr7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:50.374492 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.374447    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsr7q\" (UniqueName: \"kubernetes.io/projected/d8ef4400-3124-45b5-8a46-402b7b38f6e4-kube-api-access-vsr7q\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:49:50.374492 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.374486    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8ef4400-3124-45b5-8a46-402b7b38f6e4-extensions-socket-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:49:50.620415 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.620378    2563 generic.go:356] "Generic (PLEG): container finished" podID="d8ef4400-3124-45b5-8a46-402b7b38f6e4" containerID="fdfea039764fd0c2ea55cc6f275268877e85bddf92e03bb606bc2a21bb85d851" exitCode=0
Sep 09 20:49:50.620626 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.620428    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"
Sep 09 20:49:50.620626 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.620442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj" event={"ID":"d8ef4400-3124-45b5-8a46-402b7b38f6e4","Type":"ContainerDied","Data":"fdfea039764fd0c2ea55cc6f275268877e85bddf92e03bb606bc2a21bb85d851"}
Sep 09 20:49:50.620626 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.620497    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj" event={"ID":"d8ef4400-3124-45b5-8a46-402b7b38f6e4","Type":"ContainerDied","Data":"24f38ec259a7220ec57436aba5ab845b416523b234c0ff1401f56d7a2686d513"}
Sep 09 20:49:50.620626 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.620529    2563 scope.go:117] "RemoveContainer" containerID="fdfea039764fd0c2ea55cc6f275268877e85bddf92e03bb606bc2a21bb85d851"
Sep 09 20:49:50.630142 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.630119    2563 scope.go:117] "RemoveContainer" containerID="fdfea039764fd0c2ea55cc6f275268877e85bddf92e03bb606bc2a21bb85d851"
Sep 09 20:49:50.630399 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:49:50.630382    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfea039764fd0c2ea55cc6f275268877e85bddf92e03bb606bc2a21bb85d851\": container with ID starting with fdfea039764fd0c2ea55cc6f275268877e85bddf92e03bb606bc2a21bb85d851 not found: ID does not exist" containerID="fdfea039764fd0c2ea55cc6f275268877e85bddf92e03bb606bc2a21bb85d851"
Sep 09 20:49:50.630455 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.630408    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfea039764fd0c2ea55cc6f275268877e85bddf92e03bb606bc2a21bb85d851"} err="failed to get container status \"fdfea039764fd0c2ea55cc6f275268877e85bddf92e03bb606bc2a21bb85d851\": rpc error: code = NotFound desc = could not find container \"fdfea039764fd0c2ea55cc6f275268877e85bddf92e03bb606bc2a21bb85d851\": container with ID starting with fdfea039764fd0c2ea55cc6f275268877e85bddf92e03bb606bc2a21bb85d851 not found: ID does not exist"
Sep 09 20:49:50.642004 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.641977    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"]
Sep 09 20:49:50.645673 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:50.645649    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lxmtj"]
Sep 09 20:49:51.313399 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:51.313364    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8ef4400-3124-45b5-8a46-402b7b38f6e4" path="/var/lib/kubelet/pods/d8ef4400-3124-45b5-8a46-402b7b38f6e4/volumes"
Sep 09 20:49:55.207891 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:49:55.207843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:31.561267 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:31.561228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:50.568633 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.568597    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8"]
Sep 09 20:50:50.569107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.568908    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72218f80-ffb8-488d-9062-546bd599523f" containerName="revoke-keys"
Sep 09 20:50:50.569107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.568920    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="72218f80-ffb8-488d-9062-546bd599523f" containerName="revoke-keys"
Sep 09 20:50:50.569107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.568939    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerName="istio-proxy"
Sep 09 20:50:50.569107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.568948    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerName="istio-proxy"
Sep 09 20:50:50.569107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.568955    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8ef4400-3124-45b5-8a46-402b7b38f6e4" containerName="manager"
Sep 09 20:50:50.569107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.568960    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8ef4400-3124-45b5-8a46-402b7b38f6e4" containerName="manager"
Sep 09 20:50:50.569107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.568968    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerName="istio-proxy"
Sep 09 20:50:50.569107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.568973    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerName="istio-proxy"
Sep 09 20:50:50.569107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.569016    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8ef4400-3124-45b5-8a46-402b7b38f6e4" containerName="manager"
Sep 09 20:50:50.569107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.569026    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="72218f80-ffb8-488d-9062-546bd599523f" containerName="revoke-keys"
Sep 09 20:50:50.569107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.569035    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerName="istio-proxy"
Sep 09 20:50:50.569107 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.569043    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb1fbbc-7daf-46a3-8595-02f4f6b2a9fe" containerName="istio-proxy"
Sep 09 20:50:50.575939 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.575921    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8"
Sep 09 20:50:50.580289 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.580268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qngs9\""
Sep 09 20:50:50.586554 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.586530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8"]
Sep 09 20:50:50.677699 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.677663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffbe2b1b-5243-41dc-ad8a-b12ff45c0512-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9wjp8\" (UID: \"ffbe2b1b-5243-41dc-ad8a-b12ff45c0512\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8"
Sep 09 20:50:50.677699 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.677702    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmh8\" (UniqueName: \"kubernetes.io/projected/ffbe2b1b-5243-41dc-ad8a-b12ff45c0512-kube-api-access-5zmh8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9wjp8\" (UID: \"ffbe2b1b-5243-41dc-ad8a-b12ff45c0512\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8"
Sep 09 20:50:50.778988 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.778948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffbe2b1b-5243-41dc-ad8a-b12ff45c0512-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9wjp8\" (UID: \"ffbe2b1b-5243-41dc-ad8a-b12ff45c0512\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8"
Sep 09 20:50:50.778988 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.778987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmh8\" (UniqueName: \"kubernetes.io/projected/ffbe2b1b-5243-41dc-ad8a-b12ff45c0512-kube-api-access-5zmh8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9wjp8\" (UID: \"ffbe2b1b-5243-41dc-ad8a-b12ff45c0512\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8"
Sep 09 20:50:50.779345 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.779322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffbe2b1b-5243-41dc-ad8a-b12ff45c0512-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9wjp8\" (UID: \"ffbe2b1b-5243-41dc-ad8a-b12ff45c0512\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8"
Sep 09 20:50:50.796380 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.796345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmh8\" (UniqueName: \"kubernetes.io/projected/ffbe2b1b-5243-41dc-ad8a-b12ff45c0512-kube-api-access-5zmh8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9wjp8\" (UID: \"ffbe2b1b-5243-41dc-ad8a-b12ff45c0512\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8"
Sep 09 20:50:50.886069 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:50.885972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8"
Sep 09 20:50:51.167065 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:51.166968    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8"]
Sep 09 20:50:51.171279 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:50:51.171243    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbe2b1b_5243_41dc_ad8a_b12ff45c0512.slice/crio-cfd1b26ca14e0ac1f70b7155edeb2e0e28a641eeb1ff8b868100d8e7c6f35790 WatchSource:0}: Error finding container cfd1b26ca14e0ac1f70b7155edeb2e0e28a641eeb1ff8b868100d8e7c6f35790: Status 404 returned error can't find the container with id cfd1b26ca14e0ac1f70b7155edeb2e0e28a641eeb1ff8b868100d8e7c6f35790
Sep 09 20:50:51.893094 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:51.893058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8" event={"ID":"ffbe2b1b-5243-41dc-ad8a-b12ff45c0512","Type":"ContainerStarted","Data":"6b1a484d8696eb7ad67fcadf7f85afcd0aceb6e86648e421456394069195a178"}
Sep 09 20:50:51.893494 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:51.893117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8" event={"ID":"ffbe2b1b-5243-41dc-ad8a-b12ff45c0512","Type":"ContainerStarted","Data":"cfd1b26ca14e0ac1f70b7155edeb2e0e28a641eeb1ff8b868100d8e7c6f35790"}
Sep 09 20:50:51.893494 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:51.893157    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8"
Sep 09 20:50:51.926279 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:50:51.926231    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8" podStartSLOduration=1.926215459 podStartE2EDuration="1.926215459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:51.924143833 +0000 UTC m=+2059.163246133" watchObservedRunningTime="2026-09-09 20:50:51.926215459 +0000 UTC m=+2059.165317759"
Sep 09 20:51:01.332776 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:01.332685    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:02.899152 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:02.899119    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wjp8"
Sep 09 20:51:10.447656 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:10.447614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:10.804494 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:10.804461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:51:14.369534 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:14.369492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:14.470007 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:14.469975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:18.519436 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:18.519392    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:51:24.540822 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:24.540779    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:51:33.590806 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:33.590774    2563 scope.go:117] "RemoveContainer" containerID="2f27466902f85eb275aed9f618d7c2e98d005c1379358547e7037fe3a96c9024"
Sep 09 20:51:33.599081 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:33.599060    2563 scope.go:117] "RemoveContainer" containerID="c6b51b230a2d3f9ab5c1a991dff424d8abaa36ec5d79d357151579a7e7d4eb66"
Sep 09 20:51:33.606376 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:33.606356    2563 scope.go:117] "RemoveContainer" containerID="65694cabed7d981787cb29038fdfdfaa1264b31cbdcc311f795c47bd0de91b01"
Sep 09 20:51:36.298697 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:36.298661    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:38.817751 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:38.817716    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:51:54.951955 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:54.951919    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"]
Sep 09 20:51:54.955396 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:54.955377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:54.958639 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:54.958616    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-3d9d523f-kserve-self-signed-certs\""
Sep 09 20:51:54.968730 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:54.968703    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"]
Sep 09 20:51:54.982986 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:54.982944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:55.024022 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.023986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-tmp-dir\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.024209 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.024043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-tls-certs\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.024209 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.024060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sx59\" (UniqueName: \"kubernetes.io/projected/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-kube-api-access-7sx59\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.024209 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.024159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-home\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.024209 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.024201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-model-cache\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.024388 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.024266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-dshm\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.125250 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.125211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-dshm\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.125437 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.125285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-tmp-dir\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.125437 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.125324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-tls-certs\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.125437 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.125342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sx59\" (UniqueName: \"kubernetes.io/projected/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-kube-api-access-7sx59\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.125437 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.125371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-home\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.125437 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.125391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-model-cache\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.125792 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.125768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-tmp-dir\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.125850 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.125796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-model-cache\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.125913 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.125888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-home\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.128175 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.128148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-dshm\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.128500 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.128480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-tls-certs\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.134359 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.134281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sx59\" (UniqueName: \"kubernetes.io/projected/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-kube-api-access-7sx59\") pod \"e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") " pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.264622 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.264553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:51:55.546423 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:55.546397    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"]
Sep 09 20:51:55.549580 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:51:55.549535    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38f927c2_66a2_42d9_a6ab_ec939b3c02c6.slice/crio-ba4e268df3dea60a1ff2c6d7f95daf0930031fbad38eaaf6265491c5ebf17510 WatchSource:0}: Error finding container ba4e268df3dea60a1ff2c6d7f95daf0930031fbad38eaaf6265491c5ebf17510: Status 404 returned error can't find the container with id ba4e268df3dea60a1ff2c6d7f95daf0930031fbad38eaaf6265491c5ebf17510
Sep 09 20:51:56.176878 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:56.176834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw" event={"ID":"38f927c2-66a2-42d9-a6ab-ec939b3c02c6","Type":"ContainerStarted","Data":"08aa839a9deb650206d786343aeaa53c04c64bcec1e15f5a35b7827e3a286e43"}
Sep 09 20:51:56.177259 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:56.176889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw" event={"ID":"38f927c2-66a2-42d9-a6ab-ec939b3c02c6","Type":"ContainerStarted","Data":"ba4e268df3dea60a1ff2c6d7f95daf0930031fbad38eaaf6265491c5ebf17510"}
Sep 09 20:51:56.195380 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:56.195326    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw" podStartSLOduration=2.195308213 podStartE2EDuration="2.195308213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:56.194715462 +0000 UTC m=+2123.433817761" watchObservedRunningTime="2026-09-09 20:51:56.195308213 +0000 UTC m=+2123.434410506"
Sep 09 20:51:57.818763 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:51:57.818726    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:52:05.264894 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:05.264852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:52:05.264894 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:05.264903    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:52:05.277790 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:05.277756    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:52:06.235249 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:06.235219    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:52:33.631488 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:33.631413    2563 scope.go:117] "RemoveContainer" containerID="3775ddcb4616faef67ae499309d2dcb980065d97ec19f1e7352bbc7493a87bf2"
Sep 09 20:52:33.639338 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:33.639311    2563 scope.go:117] "RemoveContainer" containerID="4c1e02d70a43da7de4750d87e977f772f253546f712a685f6ca662b45730615c"
Sep 09 20:52:38.590684 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:38.590642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:40.169460 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:40.169427    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"]
Sep 09 20:52:40.279860 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:40.279829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:40.812361 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:40.812324    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:52:49.461392 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:49.461353    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:52:55.188804 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.188739    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw" podUID="38f927c2-66a2-42d9-a6ab-ec939b3c02c6" containerName="main" containerID="cri-o://08aa839a9deb650206d786343aeaa53c04c64bcec1e15f5a35b7827e3a286e43" gracePeriod=45
Sep 09 20:52:55.434681 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.434649    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:52:55.447812 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.447736    2563 generic.go:356] "Generic (PLEG): container finished" podID="38f927c2-66a2-42d9-a6ab-ec939b3c02c6" containerID="08aa839a9deb650206d786343aeaa53c04c64bcec1e15f5a35b7827e3a286e43" exitCode=0
Sep 09 20:52:55.447812 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.447792    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"
Sep 09 20:52:55.447812 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.447800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw" event={"ID":"38f927c2-66a2-42d9-a6ab-ec939b3c02c6","Type":"ContainerDied","Data":"08aa839a9deb650206d786343aeaa53c04c64bcec1e15f5a35b7827e3a286e43"}
Sep 09 20:52:55.448055 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.447832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw" event={"ID":"38f927c2-66a2-42d9-a6ab-ec939b3c02c6","Type":"ContainerDied","Data":"ba4e268df3dea60a1ff2c6d7f95daf0930031fbad38eaaf6265491c5ebf17510"}
Sep 09 20:52:55.448055 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.447848    2563 scope.go:117] "RemoveContainer" containerID="08aa839a9deb650206d786343aeaa53c04c64bcec1e15f5a35b7827e3a286e43"
Sep 09 20:52:55.456424 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.456406    2563 scope.go:117] "RemoveContainer" containerID="08aa839a9deb650206d786343aeaa53c04c64bcec1e15f5a35b7827e3a286e43"
Sep 09 20:52:55.456788 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:52:55.456763    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08aa839a9deb650206d786343aeaa53c04c64bcec1e15f5a35b7827e3a286e43\": container with ID starting with 08aa839a9deb650206d786343aeaa53c04c64bcec1e15f5a35b7827e3a286e43 not found: ID does not exist" containerID="08aa839a9deb650206d786343aeaa53c04c64bcec1e15f5a35b7827e3a286e43"
Sep 09 20:52:55.456908 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.456800    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08aa839a9deb650206d786343aeaa53c04c64bcec1e15f5a35b7827e3a286e43"} err="failed to get container status \"08aa839a9deb650206d786343aeaa53c04c64bcec1e15f5a35b7827e3a286e43\": rpc error: code = NotFound desc = could not find container \"08aa839a9deb650206d786343aeaa53c04c64bcec1e15f5a35b7827e3a286e43\": container with ID starting with 08aa839a9deb650206d786343aeaa53c04c64bcec1e15f5a35b7827e3a286e43 not found: ID does not exist"
Sep 09 20:52:55.571317 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.571283    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-home\") pod \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") "
Sep 09 20:52:55.571317 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.571326    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-dshm\") pod \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") "
Sep 09 20:52:55.571529 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.571382    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sx59\" (UniqueName: \"kubernetes.io/projected/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-kube-api-access-7sx59\") pod \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") "
Sep 09 20:52:55.571529 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.571415    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-tls-certs\") pod \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") "
Sep 09 20:52:55.571529 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.571448    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-model-cache\") pod \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") "
Sep 09 20:52:55.571702 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.571561    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-home" (OuterVolumeSpecName: "home") pod "38f927c2-66a2-42d9-a6ab-ec939b3c02c6" (UID: "38f927c2-66a2-42d9-a6ab-ec939b3c02c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:55.571702 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.571657    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-tmp-dir\") pod \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\" (UID: \"38f927c2-66a2-42d9-a6ab-ec939b3c02c6\") "
Sep 09 20:52:55.571804 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.571708    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "38f927c2-66a2-42d9-a6ab-ec939b3c02c6" (UID: "38f927c2-66a2-42d9-a6ab-ec939b3c02c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:55.571854 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.571805    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38f927c2-66a2-42d9-a6ab-ec939b3c02c6" (UID: "38f927c2-66a2-42d9-a6ab-ec939b3c02c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:55.571974 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.571957    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-model-cache\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:52:55.572037 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.571978    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-tmp-dir\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:52:55.572037 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.571991    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-home\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:52:55.573637 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.573605    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38f927c2-66a2-42d9-a6ab-ec939b3c02c6" (UID: "38f927c2-66a2-42d9-a6ab-ec939b3c02c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:52:55.573748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.573633    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-dshm" (OuterVolumeSpecName: "dshm") pod "38f927c2-66a2-42d9-a6ab-ec939b3c02c6" (UID: "38f927c2-66a2-42d9-a6ab-ec939b3c02c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:55.573748 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.573711    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-kube-api-access-7sx59" (OuterVolumeSpecName: "kube-api-access-7sx59") pod "38f927c2-66a2-42d9-a6ab-ec939b3c02c6" (UID: "38f927c2-66a2-42d9-a6ab-ec939b3c02c6"). InnerVolumeSpecName "kube-api-access-7sx59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:52:55.673126 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.673085    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-dshm\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:52:55.673126 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.673119    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sx59\" (UniqueName: \"kubernetes.io/projected/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-kube-api-access-7sx59\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:52:55.673126 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.673130    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f927c2-66a2-42d9-a6ab-ec939b3c02c6-tls-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:52:55.777003 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:52:55.776962    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38f927c2_66a2_42d9_a6ab_ec939b3c02c6.slice/crio-ba4e268df3dea60a1ff2c6d7f95daf0930031fbad38eaaf6265491c5ebf17510\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38f927c2_66a2_42d9_a6ab_ec939b3c02c6.slice\": RecentStats: unable to find data in memory cache]"
Sep 09 20:52:55.777171 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.777003    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"]
Sep 09 20:52:55.785539 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:55.785515    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-3d9d523f-kserve-6564765567-rxmhw"]
Sep 09 20:52:56.907508 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:56.907471    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:52:57.312797 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:52:57.312764    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38f927c2-66a2-42d9-a6ab-ec939b3c02c6" path="/var/lib/kubelet/pods/38f927c2-66a2-42d9-a6ab-ec939b3c02c6/volumes"
Sep 09 20:53:08.488078 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.488042    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"]
Sep 09 20:53:08.488488 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.488335    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f927c2-66a2-42d9-a6ab-ec939b3c02c6" containerName="main"
Sep 09 20:53:08.488488 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.488346    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f927c2-66a2-42d9-a6ab-ec939b3c02c6" containerName="main"
Sep 09 20:53:08.488488 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.488406    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f927c2-66a2-42d9-a6ab-ec939b3c02c6" containerName="main"
Sep 09 20:53:08.491020 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.491003    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.495434 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.495404    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 20:53:08.495622 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.495451    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 20:53:08.495622 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.495466    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bbx7f\""
Sep 09 20:53:08.495622 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.495409    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 20:53:08.495622 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.495538    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:53:08.511177 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.511144    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"]
Sep 09 20:53:08.515081 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.515052    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"]
Sep 09 20:53:08.518411 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.518386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:08.534857 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.534789    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"]
Sep 09 20:53:08.573967 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.573929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf4525a-317a-4234-9e99-80ab1a382c06-metrics-certs\") pod \"maas-api-5c576cc74c-bt5q4\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.573967 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.573977    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf4525a-317a-4234-9e99-80ab1a382c06-openshift-service-ca\") pod \"maas-api-5c576cc74c-bt5q4\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.574192 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.573997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jhfx\" (UniqueName: \"kubernetes.io/projected/7bf4525a-317a-4234-9e99-80ab1a382c06-kube-api-access-5jhfx\") pod \"maas-api-5c576cc74c-bt5q4\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.574192 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.574012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8676e01a-9a6e-4b9f-8a98-5139eb3247dd-openshift-service-ca\") pod \"maas-api-6bc9c6c88c-7rcjx\" (UID: \"8676e01a-9a6e-4b9f-8a98-5139eb3247dd\") " pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:08.574192 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.574069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdlvj\" (UniqueName: \"kubernetes.io/projected/8676e01a-9a6e-4b9f-8a98-5139eb3247dd-kube-api-access-qdlvj\") pod \"maas-api-6bc9c6c88c-7rcjx\" (UID: \"8676e01a-9a6e-4b9f-8a98-5139eb3247dd\") " pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:08.574192 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.574087    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8676e01a-9a6e-4b9f-8a98-5139eb3247dd-metrics-certs\") pod \"maas-api-6bc9c6c88c-7rcjx\" (UID: \"8676e01a-9a6e-4b9f-8a98-5139eb3247dd\") " pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:08.574192 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.574144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf4525a-317a-4234-9e99-80ab1a382c06-maas-api-tls\") pod \"maas-api-5c576cc74c-bt5q4\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.574356 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.574195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8676e01a-9a6e-4b9f-8a98-5139eb3247dd-maas-api-tls\") pod \"maas-api-6bc9c6c88c-7rcjx\" (UID: \"8676e01a-9a6e-4b9f-8a98-5139eb3247dd\") " pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:08.622632 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.622593    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:53:08.675287 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.675238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8676e01a-9a6e-4b9f-8a98-5139eb3247dd-maas-api-tls\") pod \"maas-api-6bc9c6c88c-7rcjx\" (UID: \"8676e01a-9a6e-4b9f-8a98-5139eb3247dd\") " pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:08.675287 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.675293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf4525a-317a-4234-9e99-80ab1a382c06-metrics-certs\") pod \"maas-api-5c576cc74c-bt5q4\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.675536 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.675415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf4525a-317a-4234-9e99-80ab1a382c06-openshift-service-ca\") pod \"maas-api-5c576cc74c-bt5q4\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.675536 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.675451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhfx\" (UniqueName: \"kubernetes.io/projected/7bf4525a-317a-4234-9e99-80ab1a382c06-kube-api-access-5jhfx\") pod \"maas-api-5c576cc74c-bt5q4\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.675536 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.675468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8676e01a-9a6e-4b9f-8a98-5139eb3247dd-openshift-service-ca\") pod \"maas-api-6bc9c6c88c-7rcjx\" (UID: \"8676e01a-9a6e-4b9f-8a98-5139eb3247dd\") " pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:08.675536 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.675495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdlvj\" (UniqueName: \"kubernetes.io/projected/8676e01a-9a6e-4b9f-8a98-5139eb3247dd-kube-api-access-qdlvj\") pod \"maas-api-6bc9c6c88c-7rcjx\" (UID: \"8676e01a-9a6e-4b9f-8a98-5139eb3247dd\") " pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:08.675536 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.675514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8676e01a-9a6e-4b9f-8a98-5139eb3247dd-metrics-certs\") pod \"maas-api-6bc9c6c88c-7rcjx\" (UID: \"8676e01a-9a6e-4b9f-8a98-5139eb3247dd\") " pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:08.675871 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.675851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf4525a-317a-4234-9e99-80ab1a382c06-maas-api-tls\") pod \"maas-api-5c576cc74c-bt5q4\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.676268 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.676243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf4525a-317a-4234-9e99-80ab1a382c06-openshift-service-ca\") pod \"maas-api-5c576cc74c-bt5q4\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.676414 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.676280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8676e01a-9a6e-4b9f-8a98-5139eb3247dd-openshift-service-ca\") pod \"maas-api-6bc9c6c88c-7rcjx\" (UID: \"8676e01a-9a6e-4b9f-8a98-5139eb3247dd\") " pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:08.678216 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.678164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf4525a-317a-4234-9e99-80ab1a382c06-metrics-certs\") pod \"maas-api-5c576cc74c-bt5q4\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.678324 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.678254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8676e01a-9a6e-4b9f-8a98-5139eb3247dd-maas-api-tls\") pod \"maas-api-6bc9c6c88c-7rcjx\" (UID: \"8676e01a-9a6e-4b9f-8a98-5139eb3247dd\") " pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:08.678522 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.678499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf4525a-317a-4234-9e99-80ab1a382c06-maas-api-tls\") pod \"maas-api-5c576cc74c-bt5q4\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.678568 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.678500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8676e01a-9a6e-4b9f-8a98-5139eb3247dd-metrics-certs\") pod \"maas-api-6bc9c6c88c-7rcjx\" (UID: \"8676e01a-9a6e-4b9f-8a98-5139eb3247dd\") " pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:08.685541 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.685510    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhfx\" (UniqueName: \"kubernetes.io/projected/7bf4525a-317a-4234-9e99-80ab1a382c06-kube-api-access-5jhfx\") pod \"maas-api-5c576cc74c-bt5q4\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.685674 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.685615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdlvj\" (UniqueName: \"kubernetes.io/projected/8676e01a-9a6e-4b9f-8a98-5139eb3247dd-kube-api-access-qdlvj\") pod \"maas-api-6bc9c6c88c-7rcjx\" (UID: \"8676e01a-9a6e-4b9f-8a98-5139eb3247dd\") " pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:08.801264 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.801166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:08.828098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:08.828061    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:09.086907 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:09.086754    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"]
Sep 09 20:53:09.089667 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:53:09.089627    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf4525a_317a_4234_9e99_80ab1a382c06.slice/crio-7a8f23bd42d8f27a61d1d05a14bc15f8bb621185a3aa5b0fcb9cf35c8f67d4b0 WatchSource:0}: Error finding container 7a8f23bd42d8f27a61d1d05a14bc15f8bb621185a3aa5b0fcb9cf35c8f67d4b0: Status 404 returned error can't find the container with id 7a8f23bd42d8f27a61d1d05a14bc15f8bb621185a3aa5b0fcb9cf35c8f67d4b0
Sep 09 20:53:09.091171 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:09.091153    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:53:09.128419 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:53:09.128393    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8676e01a_9a6e_4b9f_8a98_5139eb3247dd.slice/crio-6e2ba78281d4beacf98781cf51de9f7b79de736a1ead89f377856d7d219b46bc WatchSource:0}: Error finding container 6e2ba78281d4beacf98781cf51de9f7b79de736a1ead89f377856d7d219b46bc: Status 404 returned error can't find the container with id 6e2ba78281d4beacf98781cf51de9f7b79de736a1ead89f377856d7d219b46bc
Sep 09 20:53:09.128537 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:09.128487    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"]
Sep 09 20:53:09.517839 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:09.517798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx" event={"ID":"8676e01a-9a6e-4b9f-8a98-5139eb3247dd","Type":"ContainerStarted","Data":"6e2ba78281d4beacf98781cf51de9f7b79de736a1ead89f377856d7d219b46bc"}
Sep 09 20:53:09.518922 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:09.518895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4" event={"ID":"7bf4525a-317a-4234-9e99-80ab1a382c06","Type":"ContainerStarted","Data":"7a8f23bd42d8f27a61d1d05a14bc15f8bb621185a3aa5b0fcb9cf35c8f67d4b0"}
Sep 09 20:53:10.524349 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:10.524313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4" event={"ID":"7bf4525a-317a-4234-9e99-80ab1a382c06","Type":"ContainerStarted","Data":"b9254efa30a0d2245781a6688f449002aec1fb98daa6b0702dbc2a3a54e75fc5"}
Sep 09 20:53:10.524796 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:10.524380    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:10.525923 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:10.525896    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx" event={"ID":"8676e01a-9a6e-4b9f-8a98-5139eb3247dd","Type":"ContainerStarted","Data":"2d39ea864e21f324e44ac2b710fc97a89496b1e6af4a9e899055c0a8b480ad82"}
Sep 09 20:53:10.526043 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:10.526016    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:10.550965 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:10.550908    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4" podStartSLOduration=1.983438142 podStartE2EDuration="2.550891438s" totalImagesPullingTime="567.453296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:10.550151065 +0000 UTC m=+2197.789253365" watchObservedRunningTime="2026-09-09 20:53:10.550891438 +0000 UTC m=+2197.789993737"
Sep 09 20:53:10.570641 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:10.570563    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx" podStartSLOduration=2.015539874 podStartE2EDuration="2.570541921s" totalImagesPullingTime="555.002047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:10.569256799 +0000 UTC m=+2197.808359100" watchObservedRunningTime="2026-09-09 20:53:10.570541921 +0000 UTC m=+2197.809644226"
Sep 09 20:53:14.718755 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:14.718719    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:53:16.535794 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:16.535760    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6bc9c6c88c-7rcjx"
Sep 09 20:53:16.536244 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:16.536134    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:16.615878 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:16.615840    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"]
Sep 09 20:53:16.616055 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:16.616038    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4" podUID="7bf4525a-317a-4234-9e99-80ab1a382c06" containerName="maas-api" containerID="cri-o://b9254efa30a0d2245781a6688f449002aec1fb98daa6b0702dbc2a3a54e75fc5" gracePeriod=30
Sep 09 20:53:16.862258 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:16.862224    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:16.944994 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:16.944956    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf4525a-317a-4234-9e99-80ab1a382c06-openshift-service-ca\") pod \"7bf4525a-317a-4234-9e99-80ab1a382c06\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") "
Sep 09 20:53:16.945190 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:16.945040    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf4525a-317a-4234-9e99-80ab1a382c06-metrics-certs\") pod \"7bf4525a-317a-4234-9e99-80ab1a382c06\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") "
Sep 09 20:53:16.945190 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:16.945078    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf4525a-317a-4234-9e99-80ab1a382c06-maas-api-tls\") pod \"7bf4525a-317a-4234-9e99-80ab1a382c06\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") "
Sep 09 20:53:16.945190 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:16.945123    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jhfx\" (UniqueName: \"kubernetes.io/projected/7bf4525a-317a-4234-9e99-80ab1a382c06-kube-api-access-5jhfx\") pod \"7bf4525a-317a-4234-9e99-80ab1a382c06\" (UID: \"7bf4525a-317a-4234-9e99-80ab1a382c06\") "
Sep 09 20:53:16.945458 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:16.945432    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf4525a-317a-4234-9e99-80ab1a382c06-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7bf4525a-317a-4234-9e99-80ab1a382c06" (UID: "7bf4525a-317a-4234-9e99-80ab1a382c06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:53:16.947299 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:16.947262    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf4525a-317a-4234-9e99-80ab1a382c06-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7bf4525a-317a-4234-9e99-80ab1a382c06" (UID: "7bf4525a-317a-4234-9e99-80ab1a382c06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:53:16.947411 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:16.947323    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf4525a-317a-4234-9e99-80ab1a382c06-kube-api-access-5jhfx" (OuterVolumeSpecName: "kube-api-access-5jhfx") pod "7bf4525a-317a-4234-9e99-80ab1a382c06" (UID: "7bf4525a-317a-4234-9e99-80ab1a382c06"). InnerVolumeSpecName "kube-api-access-5jhfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:16.947516 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:16.947502    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf4525a-317a-4234-9e99-80ab1a382c06-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7bf4525a-317a-4234-9e99-80ab1a382c06" (UID: "7bf4525a-317a-4234-9e99-80ab1a382c06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:53:17.046119 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:17.046075    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf4525a-317a-4234-9e99-80ab1a382c06-metrics-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:53:17.046119 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:17.046108    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf4525a-317a-4234-9e99-80ab1a382c06-maas-api-tls\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:53:17.046119 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:17.046118    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jhfx\" (UniqueName: \"kubernetes.io/projected/7bf4525a-317a-4234-9e99-80ab1a382c06-kube-api-access-5jhfx\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:53:17.046119 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:17.046127    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf4525a-317a-4234-9e99-80ab1a382c06-openshift-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\""
Sep 09 20:53:17.559346 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:17.559307    2563 generic.go:356] "Generic (PLEG): container finished" podID="7bf4525a-317a-4234-9e99-80ab1a382c06" containerID="b9254efa30a0d2245781a6688f449002aec1fb98daa6b0702dbc2a3a54e75fc5" exitCode=0
Sep 09 20:53:17.559346 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:17.559352    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"
Sep 09 20:53:17.559926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:17.559394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4" event={"ID":"7bf4525a-317a-4234-9e99-80ab1a382c06","Type":"ContainerDied","Data":"b9254efa30a0d2245781a6688f449002aec1fb98daa6b0702dbc2a3a54e75fc5"}
Sep 09 20:53:17.559926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:17.559441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4" event={"ID":"7bf4525a-317a-4234-9e99-80ab1a382c06","Type":"ContainerDied","Data":"7a8f23bd42d8f27a61d1d05a14bc15f8bb621185a3aa5b0fcb9cf35c8f67d4b0"}
Sep 09 20:53:17.559926 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:17.559465    2563 scope.go:117] "RemoveContainer" containerID="b9254efa30a0d2245781a6688f449002aec1fb98daa6b0702dbc2a3a54e75fc5"
Sep 09 20:53:17.568316 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:17.568295    2563 scope.go:117] "RemoveContainer" containerID="b9254efa30a0d2245781a6688f449002aec1fb98daa6b0702dbc2a3a54e75fc5"
Sep 09 20:53:17.568620 ip-10-0-143-241 kubenswrapper[2563]: E0909 20:53:17.568601    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9254efa30a0d2245781a6688f449002aec1fb98daa6b0702dbc2a3a54e75fc5\": container with ID starting with b9254efa30a0d2245781a6688f449002aec1fb98daa6b0702dbc2a3a54e75fc5 not found: ID does not exist" containerID="b9254efa30a0d2245781a6688f449002aec1fb98daa6b0702dbc2a3a54e75fc5"
Sep 09 20:53:17.568699 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:17.568633    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9254efa30a0d2245781a6688f449002aec1fb98daa6b0702dbc2a3a54e75fc5"} err="failed to get container status \"b9254efa30a0d2245781a6688f449002aec1fb98daa6b0702dbc2a3a54e75fc5\": rpc error: code = NotFound desc = could not find container \"b9254efa30a0d2245781a6688f449002aec1fb98daa6b0702dbc2a3a54e75fc5\": container with ID starting with b9254efa30a0d2245781a6688f449002aec1fb98daa6b0702dbc2a3a54e75fc5 not found: ID does not exist"
Sep 09 20:53:17.576775 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:17.576710    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"]
Sep 09 20:53:17.584217 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:17.584179    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c576cc74c-bt5q4"]
Sep 09 20:53:19.312920 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:19.312886    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bf4525a-317a-4234-9e99-80ab1a382c06" path="/var/lib/kubelet/pods/7bf4525a-317a-4234-9e99-80ab1a382c06/volumes"
Sep 09 20:53:22.602201 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:22.602170    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:53:28.719597 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:28.719542    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:53:33.659443 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:33.659404    2563 scope.go:117] "RemoveContainer" containerID="6f2a64c024594cf74960a6e4b68708c17c15394b50fdeb54e810169ff5231b75"
Sep 09 20:53:35.630650 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:35.630611    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8gkmm"]
Sep 09 20:53:44.723234 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:44.723200    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-z8k8d_c1fa4292-059b-40a2-a8ca-86364d7f8033/manager/0.log"
Sep 09 20:53:44.858098 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:44.858068    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dbdbfdc7-7n27t_db07ac0a-97c1-415a-b685-37f8e342af6f/copy-manifests/0.log"
Sep 09 20:53:44.870059 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:44.870027    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dbdbfdc7-7n27t_db07ac0a-97c1-415a-b685-37f8e342af6f/manager/0.log"
Sep 09 20:53:45.532215 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:45.532177    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6bc9c6c88c-7rcjx_8676e01a-9a6e-4b9f-8a98-5139eb3247dd/maas-api/0.log"
Sep 09 20:53:45.770326 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:45.770289    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816430-f8b4k_b0e24d8b-8b52-4b62-8419-a82b3b7cdf16/cleanup/0.log"
Sep 09 20:53:45.885960 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:45.885882    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816445-2c4jr_4ef979d8-f5ac-48e7-8216-10222766c6a4/cleanup/0.log"
Sep 09 20:53:46.929360 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:46.929327    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-zjkjd_efcef3f7-8aea-4cb6-8b9f-e05ad436ab52/manager/0.log"
Sep 09 20:53:47.359084 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:47.359046    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-9wjp8_ffbe2b1b-5243-41dc-ad8a-b12ff45c0512/manager/0.log"
Sep 09 20:53:47.482074 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:47.482041    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8gkmm_2dbc1335-4b11-4ff0-a531-35d43b20dbb3/limitador/0.log"
Sep 09 20:53:47.963518 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:47.963486    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5_45323d30-028c-479f-8086-28e773041b10/istio-proxy/0.log"
Sep 09 20:53:48.089843 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:48.089811    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qqz4k_3401c08b-6b18-4a85-94b7-5b07118d346b/discovery/0.log"
Sep 09 20:53:48.316815 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:48.316732    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-758cfc744b-w7r8z_03d79687-ff1c-49fb-ae29-37e8901a5e68/kube-auth-proxy/0.log"
Sep 09 20:53:49.596104 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:49.596065    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl98xk_e909a0fa-cce3-4f44-aca4-08bad6e0afe9/main/0.log"
Sep 09 20:53:55.749681 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:55.749595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:56.032694 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:53:56.032590    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:00.629307 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:00.629274    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lsr8k_d65fd73a-56b3-4e21-988d-cd2d26bb8847/global-pull-secret-syncer/0.log"
Sep 09 20:54:00.837723 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:00.837690    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jx5cd_73d9e6c4-c63a-458e-a181-def2c4405b45/konnectivity-agent/0.log"
Sep 09 20:54:01.307927 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:01.307895    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-241.ec2.internal_20ada389393df0af1b24f49af160350a/haproxy/0.log"
Sep 09 20:54:05.252058 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:05.252027    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-zjkjd_efcef3f7-8aea-4cb6-8b9f-e05ad436ab52/manager/0.log"
Sep 09 20:54:05.495186 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:05.495157    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-9wjp8_ffbe2b1b-5243-41dc-ad8a-b12ff45c0512/manager/0.log"
Sep 09 20:54:05.598764 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:05.598684    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8gkmm_2dbc1335-4b11-4ff0-a531-35d43b20dbb3/limitador/0.log"
Sep 09 20:54:07.830649 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:07.830619    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7sfj_1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98/node-exporter/0.log"
Sep 09 20:54:07.859311 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:07.859281    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7sfj_1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98/kube-rbac-proxy/0.log"
Sep 09 20:54:07.890941 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:07.890913    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7sfj_1e8fb8cd-6fbe-4b8a-bce0-8c27675afd98/init-textfile/0.log"
Sep 09 20:54:10.052402 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.052364    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"]
Sep 09 20:54:10.052810 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.052694    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf4525a-317a-4234-9e99-80ab1a382c06" containerName="maas-api"
Sep 09 20:54:10.052810 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.052708    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf4525a-317a-4234-9e99-80ab1a382c06" containerName="maas-api"
Sep 09 20:54:10.052810 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.052767    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf4525a-317a-4234-9e99-80ab1a382c06" containerName="maas-api"
Sep 09 20:54:10.055685 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.055664    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.058243 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.058219    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mp9c5\"/\"openshift-service-ca.crt\""
Sep 09 20:54:10.058368 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.058268    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mp9c5\"/\"kube-root-ca.crt\""
Sep 09 20:54:10.059282 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.059264    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mp9c5\"/\"default-dockercfg-j5dnh\""
Sep 09 20:54:10.070822 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.070798    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"]
Sep 09 20:54:10.091548 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.091513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-lib-modules\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.091724 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.091598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-proc\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.091724 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.091623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-podres\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.091724 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.091664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pcvd\" (UniqueName: \"kubernetes.io/projected/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-kube-api-access-4pcvd\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.091828 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.091718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-sys\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.193042 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.192999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-proc\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.193042 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.193047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-podres\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.193265 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.193100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pcvd\" (UniqueName: \"kubernetes.io/projected/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-kube-api-access-4pcvd\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.193265 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.193136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-sys\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.193265 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.193181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-lib-modules\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.193265 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.193182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-sys\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.193265 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.193136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-proc\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.193506 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.193268    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-podres\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.193506 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.193307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-lib-modules\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.202398 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.202360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pcvd\" (UniqueName: \"kubernetes.io/projected/65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef-kube-api-access-4pcvd\") pod \"perf-node-gather-daemonset-n2dxx\" (UID: \"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.365411 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.365313    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.639585 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.639512    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"]
Sep 09 20:54:10.642355 ip-10-0-143-241 kubenswrapper[2563]: W0909 20:54:10.642323    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod65d64e8d_26fc_4090_a6dc_dc0c44a4b2ef.slice/crio-ba2f4f9a98f13f47160bf06884f83bced932e99b62f7c3ebab9dd49ded0bcd1d WatchSource:0}: Error finding container ba2f4f9a98f13f47160bf06884f83bced932e99b62f7c3ebab9dd49ded0bcd1d: Status 404 returned error can't find the container with id ba2f4f9a98f13f47160bf06884f83bced932e99b62f7c3ebab9dd49ded0bcd1d
Sep 09 20:54:10.795323 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.795279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx" event={"ID":"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef","Type":"ContainerStarted","Data":"25108baa91fae50f79f26231f4190a008a5dff32b04734db0972916c640a7510"}
Sep 09 20:54:10.795504 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.795346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx" event={"ID":"65d64e8d-26fc-4090-a6dc-dc0c44a4b2ef","Type":"ContainerStarted","Data":"ba2f4f9a98f13f47160bf06884f83bced932e99b62f7c3ebab9dd49ded0bcd1d"}
Sep 09 20:54:10.795504 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.795374    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:10.815086 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:10.815030    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx" podStartSLOduration=0.81501506 podStartE2EDuration="815.01506ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:10.81368824 +0000 UTC m=+2258.052790540" watchObservedRunningTime="2026-09-09 20:54:10.81501506 +0000 UTC m=+2258.054117360"
Sep 09 20:54:13.185964 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:13.185925    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ng7mg_e51731de-fe8e-4ea7-a1f9-7661ebab8c1a/dns/0.log"
Sep 09 20:54:13.224526 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:13.224500    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ng7mg_e51731de-fe8e-4ea7-a1f9-7661ebab8c1a/kube-rbac-proxy/0.log"
Sep 09 20:54:13.403724 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:13.403699    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zsfbn_1833449f-dfe7-4c13-ae92-562b409d1cf6/dns-node-resolver/0.log"
Sep 09 20:54:13.978563 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:13.978525    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84d69c5574-drmrj_e592dada-173e-4c93-aa67-53e16ab500c7/registry/0.log"
Sep 09 20:54:14.039617 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:14.039582    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t64d7_36251d92-c8c9-450d-ac2c-103c4d5d38e6/node-ca/0.log"
Sep 09 20:54:15.073163 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:15.073131    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5_45323d30-028c-479f-8086-28e773041b10/istio-proxy/0.log"
Sep 09 20:54:15.641911 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:15.641876    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qqz4k_3401c08b-6b18-4a85-94b7-5b07118d346b/discovery/0.log"
Sep 09 20:54:15.694755 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:15.694725    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-758cfc744b-w7r8z_03d79687-ff1c-49fb-ae29-37e8901a5e68/kube-auth-proxy/0.log"
Sep 09 20:54:16.808411 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:16.808385    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-n2dxx"
Sep 09 20:54:16.899397 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:16.899362    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xvxgm_d3946c37-c56f-44b5-82ab-5a1b617fb2b1/serve-healthcheck-canary/0.log"
Sep 09 20:54:17.511013 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:17.510986    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dcf7r_4d384213-5569-4654-bb4f-6e1834675c73/kube-rbac-proxy/0.log"
Sep 09 20:54:17.541378 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:17.541341    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dcf7r_4d384213-5569-4654-bb4f-6e1834675c73/exporter/0.log"
Sep 09 20:54:17.570256 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:17.570225    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dcf7r_4d384213-5569-4654-bb4f-6e1834675c73/extractor/0.log"
Sep 09 20:54:20.723404 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:20.723353    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5_45323d30-028c-479f-8086-28e773041b10/istio-proxy/0.log"
Sep 09 20:54:20.724912 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:20.724890    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-z8k8d_c1fa4292-059b-40a2-a8ca-86364d7f8033/manager/0.log"
Sep 09 20:54:20.755407 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:20.755374    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dbdbfdc7-7n27t_db07ac0a-97c1-415a-b685-37f8e342af6f/manager/0.log"
Sep 09 20:54:20.801793 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:20.801754    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dbdbfdc7-7n27t_db07ac0a-97c1-415a-b685-37f8e342af6f/copy-manifests/0.log"
Sep 09 20:54:21.275002 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:21.274970    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qqz4k_3401c08b-6b18-4a85-94b7-5b07118d346b/discovery/0.log"
Sep 09 20:54:21.334069 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:21.334031    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-758cfc744b-w7r8z_03d79687-ff1c-49fb-ae29-37e8901a5e68/kube-auth-proxy/0.log"
Sep 09 20:54:22.918584 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:22.918527    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-t5vb5_45323d30-028c-479f-8086-28e773041b10/istio-proxy/0.log"
Sep 09 20:54:23.393727 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:23.393689    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qqz4k_3401c08b-6b18-4a85-94b7-5b07118d346b/discovery/0.log"
Sep 09 20:54:23.456789 ip-10-0-143-241 kubenswrapper[2563]: I0909 20:54:23.456747    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-758cfc744b-w7r8z_03d79687-ff1c-49fb-ae29-37e8901a5e68/kube-auth-proxy/0.log"