Sep 15 13:36:27.865566 ip-10-0-138-252 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 13:36:27.865578 ip-10-0-138-252 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 13:36:27.865587 ip-10-0-138-252 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 13:36:27.865974 ip-10-0-138-252 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 13:36:38.021175 ip-10-0-138-252 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 13:36:38.021189 ip-10-0-138-252 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 05ac900a85204c20850dd47f6c1dcf11 --
Sep 15 13:38:20.361342 ip-10-0-138-252 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 13:38:20.747919 ip-10-0-138-252 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 13:38:20.747919 ip-10-0-138-252 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 13:38:20.747919 ip-10-0-138-252 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 13:38:20.747919 ip-10-0-138-252 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 13:38:20.747919 ip-10-0-138-252 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 13:38:20.749082 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.748978    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 13:38:20.753608 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753589    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 13:38:20.753608 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753608    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753613    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753617    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753620    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753624    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753627    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753636    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753639    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753644    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753647    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753650    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753653    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753656    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753658    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753661    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753663    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753667    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753669    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753672    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753675    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 13:38:20.753677 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753679    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753682    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753686    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753689    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753691    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753694    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753697    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753699    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753702    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753704    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753707    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753709    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753712    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753715    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753719    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753723    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753726    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753730    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753735    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 13:38:20.754270 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753738    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753741    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753745    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753747    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753751    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753753    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753755    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753759    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753762    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753764    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753767    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753770    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753772    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753775    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753777    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753781    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753783    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753786    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753789    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753792    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 13:38:20.754962 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753794    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753797    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753799    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753802    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753804    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753807    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753809    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753812    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753814    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753817    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753819    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753823    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753827    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753829    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753832    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753835    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753838    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753840    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753843    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 13:38:20.755480 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753846    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753849    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753851    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753854    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753856    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753859    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753861    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753864    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753868    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753870    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753873    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753876    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753878    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753881    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753883    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753886    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753888    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753891    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753893    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 13:38:20.755935 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.753896    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756083    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756096    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756099    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756103    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756107    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756109    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756113    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756119    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756122    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756125    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756130    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756134    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756137    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756141    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756144    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756147    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756150    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756153    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756155    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 13:38:20.756420 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756158    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756164    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756167    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756170    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756172    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756175    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756178    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756181    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756184    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756186    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756190    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756194    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756197    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756203    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756207    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756210    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756213    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756215    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756218    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756222    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 13:38:20.756951 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756225    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756228    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756230    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756233    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756236    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756238    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756243    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756246    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756248    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756251    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756254    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756256    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756259    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756261    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756264    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756267    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756269    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756272    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756275    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 13:38:20.757469 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756281    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756283    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756286    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756288    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756291    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756295    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756298    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756301    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756304    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756306    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756309    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756312    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756317    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756320    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756323    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756326    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756328    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756331    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756334    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756336    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 13:38:20.757937 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756339    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756342    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756344    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756347    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756352    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756355    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756357    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756361    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756365    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756369    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756372    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756375    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756378    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756381    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756383    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756389    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756392    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756394    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756397    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 13:38:20.758444 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756400    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.756403    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757053    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757060    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757063    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757066    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757069    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757072    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757078    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757081    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757084    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757087    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757090    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757093    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757096    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757100    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757104    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757107    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757110    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757114    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 13:38:20.758910 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757119    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757122    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757125    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757128    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757132    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757135    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757138    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757142    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757145    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757148    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757152    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757154    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757160    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757164    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757167    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757169    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757172    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757175    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757178    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 13:38:20.759400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757180    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757184    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757187    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757190    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757193    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757198    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757201    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757204    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757206    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757209    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757212    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757215    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757218    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757222    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757225    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757228    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757230    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757236    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757238    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 13:38:20.759854 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757241    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757292    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757315    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757334    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757417    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757423    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757428    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757431    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757434    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757437    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757439    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757442    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757444    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757447    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757449    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757453    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757456    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757459    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757461    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757466    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 13:38:20.760330 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757469    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757472    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757475    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757477    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757480    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757483    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757486    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757488    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757491    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757493    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757496    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757498    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757501    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757504    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757506    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757509    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757511    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757513    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757517    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757520    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 13:38:20.760830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757523    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757525    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.757528    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757638    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757680    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757688    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757692    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757697    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757700    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757705    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757710    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757713    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757716    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757719    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757722    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757725    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757729    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757731    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757734    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757737    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757740    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757745    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757748    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757751    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 13:38:20.761338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757754    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757758    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757762    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757765    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757768    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757772    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757775    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757778    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757782    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757785    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757788    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757792    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757795    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757798    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757800    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757803    2582 flags.go:64] FLAG: --enable-server="true"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757806    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757811    2582 flags.go:64] FLAG: --event-burst="100"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757814    2582 flags.go:64] FLAG: --event-qps="50"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757817    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757820    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757823    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757827    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757830    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757833    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 13:38:20.761923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757836    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757839    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757842    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757845    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757848    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757851    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757854    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757857    2582 flags.go:64] FLAG: --feature-gates=""
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757861    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757864    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757867    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757871    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757874    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757877    2582 flags.go:64] FLAG: --help="false"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757879    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-138-252.ec2.internal"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757883    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757886    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757888    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757892    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757895    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757898    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757900    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757903    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 13:38:20.762571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757907    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757910    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757913    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757916    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757919    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757921    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757927    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757930    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757932    2582 flags.go:64] FLAG: --lock-file=""
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757936    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757939    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757942    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757952    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757956    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757961    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757964    2582 flags.go:64] FLAG: --logging-format="text"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757967    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757971    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757974    2582 flags.go:64] FLAG: --manifest-url=""
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757977    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757981    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757984    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757988    2582 flags.go:64] FLAG: --max-pods="110"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757991    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757994    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 13:38:20.763177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.757997    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758001    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758004    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758006    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758024    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758037    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758040    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758043    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758047    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758050    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758056    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758059    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758062    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758065    2582 flags.go:64] FLAG: --port="10250"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758069    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758072    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06088536d2d607271"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758075    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758078    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758081    2582 flags.go:64] FLAG: --register-node="true"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758084    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758087    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758092    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758095    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758098    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758102    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 13:38:20.763783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758105    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758107    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758110    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758113    2582 flags.go:64] FLAG: --runonce="false"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758116    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758119    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758122    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758125    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758128    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758131    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758133    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758136    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758139    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758142    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758145    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758148    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758151    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758154    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758157    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758163    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758166    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758169    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758173    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758175    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758178    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 13:38:20.764418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758181    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758184    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758187    2582 flags.go:64] FLAG: --v="2"
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758192    2582 flags.go:64] FLAG: --version="false"
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758197    2582 flags.go:64] FLAG: --vmodule=""
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758201    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.758205    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758323    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758327    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758330    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758333    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758337    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758339    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758342    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758346    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758348    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758352    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758354    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758357    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758360    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758363    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 13:38:20.765060 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758366    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758369    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758371    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758374    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758376    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758379    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758381    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758384    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758387    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758389    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758392    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758394    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758397    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758399    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758405    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758408    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758412    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758415    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758418    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758421    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 13:38:20.765694 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758423    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758426    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758429    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758432    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758435    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758439    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758443    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758446    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758449    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758452    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758456    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758459    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758462    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758464    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758467    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758470    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758473    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758475    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758478    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 13:38:20.766284 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758481    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758483    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758486    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758488    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758491    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758494    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758496    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758502    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758504    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758507    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758510    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758512    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758515    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758517    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758520    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758522    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758525    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758528    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758530    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758533    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 13:38:20.766734 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758536    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758538    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758541    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758543    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758546    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758548    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758551    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758554    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758556    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758559    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758561    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758564    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758568    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758570    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758573    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758575    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758578    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758581    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758583    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758586    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 13:38:20.767252 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758589    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 13:38:20.767740 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758591    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 13:38:20.767740 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758594    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 13:38:20.767740 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758597    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 13:38:20.767740 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758601    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 13:38:20.767740 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.758605    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 13:38:20.767740 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.759642    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 13:38:20.767740 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.767360    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 13:38:20.767740 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.767377    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 13:38:20.768287 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.768277    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 13:38:20.768326 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.768287    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 13:38:20.768375 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768366    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 13:38:20.768375 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768374    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768378    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768382    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768385    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768388    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768391    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768394    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768397    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768399    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768402    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768405    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768408    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768410    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768413    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768416    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768420    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768424    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768428    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768430    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768434    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 13:38:20.768429 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768437    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768439    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768442    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768445    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768448    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768450    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768453    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768455    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768459    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768462    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768464    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768467    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768470    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768472    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768475    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768478    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768480    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768483    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768486    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768488    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 13:38:20.768930 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768491    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768493    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768496    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768499    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768503    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768506    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768509    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768512    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768515    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768518    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768520    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768523    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768525    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768528    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768531    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768534    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768536    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768539    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768542    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768544    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 13:38:20.769451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768547    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768550    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768555    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768558    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768561    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768564    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768567    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768570    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768573    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768576    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768578    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768581    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768583    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768586    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768589    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768592    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768594    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768597    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768599    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 13:38:20.769972 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768602    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768605    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768607    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768610    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768612    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768615    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768617    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768621    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768623    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768626    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768629    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768632    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768634    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768637    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768639    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768642    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768645    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768647    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 13:38:20.770471 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768650    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.768655    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768791    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768803    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768808    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768811    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768814    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768817    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768820    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768823    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768825    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768828    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768830    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768833    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 13:38:20.770893 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768836    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768840    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768844    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768847    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768850    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768853    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768857    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768861    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768864    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768867    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768870    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768873    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768876    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768879    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768881    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768884    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768887    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768889    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 13:38:20.771246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768892    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768895    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768897    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768900    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768903    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768905    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768908    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768910    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768913    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768915    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768918    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768920    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768923    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768925    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768928    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768931    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768934    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768937    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768939    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768942    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 13:38:20.771680 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768945    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768947    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768950    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768953    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768956    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768959    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768961    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768964    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768966    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768969    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768971    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768975    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768977    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768980    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768982    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768985    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768988    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768990    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768992    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768995    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 13:38:20.772211 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.768998    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769000    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769003    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769005    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769022    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769026    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769028    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769031    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769034    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769036    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769039    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769042    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769044    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769046    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769049    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769052    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769055    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769058    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769060    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769063    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 13:38:20.772684 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769067    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769070    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769073    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769076    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769079    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769082    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769084    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769087    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:20.769089    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.769095    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.769220    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.771687    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.772701    2582 server.go:1019] "Starting client certificate rotation"
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.772808    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 13:38:20.773209 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.772847    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 13:38:20.775603 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.775588    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 13:38:20.776961 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.776941    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 13:38:20.788931 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.788482    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 15 13:38:20.792606 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.792589    2582 log.go:25] "Validated CRI v1 image API"
Sep 15 13:38:20.793404 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.793390    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 13:38:20.796054 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.796030    2582 fs.go:135] Filesystem UUIDs: map[5ca6250e-a810-4fdf-83ba-d5148e0cd4b2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 de4b3aa4-b340-4bc3-a8d9-b5097fe37126:/dev/nvme0n1p3]
Sep 15 13:38:20.796124 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.796053    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 13:38:20.802242 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.802129    2582 manager.go:216] Machine: {Timestamp:2026-09-15 13:38:20.800681987 +0000 UTC m=+0.333714436 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104242 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec272c5409a07e9f7e768052bb9134df SystemUUID:ec272c54-09a0-7e9f-7e76-8052bb9134df BootID:05ac900a-8520-4c20-850d-d47f6c1dcf11 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:eb:3f:6a:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:eb:3f:6a:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:02:7b:38:d2:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 13:38:20.802242 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.802240    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 13:38:20.802351 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.802322    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 13:38:20.805783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.805755    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 13:38:20.805931 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.805787    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-252.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 13:38:20.806402 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.806382    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 13:38:20.807384 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.807371    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 13:38:20.807427 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.807389    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 13:38:20.807427 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.807397    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 13:38:20.808260 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.808249    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 13:38:20.811353 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.811339    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 13:38:20.816903 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.816888    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 13:38:20.817044 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.817034    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 13:38:20.817080 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.817066    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 13:38:20.817111 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.817080    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 13:38:20.817111 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.817109    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 13:38:20.817164 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.817122    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 15 13:38:20.817164 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.817131    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 13:38:20.817539 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.817528    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 13:38:20.817577 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.817553    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 13:38:20.820108 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.820092    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 13:38:20.823311 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.823295    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 13:38:20.823374 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.823329    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 13:38:20.825508 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.825493    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 13:38:20.825558 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.825513    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 13:38:20.825558 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.825520    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 13:38:20.825558 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.825525    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 13:38:20.825558 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.825534    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 13:38:20.825558 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.825539    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 13:38:20.825558 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.825553    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 13:38:20.825558 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.825559    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 13:38:20.825737 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.825566    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 13:38:20.825737 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.825573    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 13:38:20.825737 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.825582    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 13:38:20.825737 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.825587    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 13:38:20.826234 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:20.826214    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 13:38:20.826288 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:20.826236    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 13:38:20.826541 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.826529    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 13:38:20.826569 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.826543    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 13:38:20.830345 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.830333    2582 server.go:1268] "Started kubelet"
Sep 15 13:38:20.830430 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.830396    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 13:38:20.830589 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.830536    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 13:38:20.830693 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.830641    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 13:38:20.831170 ip-10-0-138-252 systemd[1]: Started Kubernetes Kubelet.
Sep 15 13:38:20.832462 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.832444    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 13:38:20.833292 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.833276    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 13:38:20.837750 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:20.837724    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 13:38:20.842093 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.842075    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-252.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 13:38:20.842958 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:20.841855    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-252.ec2.internal.18d5821ffbd55fd0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-252.ec2.internal,UID:ip-10-0-138-252.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-252.ec2.internal,},FirstTimestamp:2026-09-15 13:38:20.83030216 +0000 UTC m=+0.363334609,LastTimestamp:2026-09-15 13:38:20.83030216 +0000 UTC m=+0.363334609,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-252.ec2.internal,}"
Sep 15 13:38:20.843408 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.843392    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 13:38:20.843483 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.843402    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 13:38:20.843595 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:20.843576    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:20.844197 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.844168    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 13:38:20.844197 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.844181    2582 factory.go:55] Registering systemd factory
Sep 15 13:38:20.844197 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.844198    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 15 13:38:20.844400 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.844209    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 13:38:20.844400 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.844187    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 13:38:20.844400 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.844291    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 13:38:20.844400 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.844299    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 13:38:20.844400 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.844392    2582 factory.go:153] Registering CRI-O factory
Sep 15 13:38:20.844400 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.844405    2582 factory.go:223] Registration of the crio container factory successfully
Sep 15 13:38:20.844675 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.844452    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 13:38:20.844675 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.844470    2582 factory.go:103] Registering Raw factory
Sep 15 13:38:20.844675 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.844487    2582 manager.go:1196] Started watching for new ooms in manager
Sep 15 13:38:20.844859 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.844848    2582 manager.go:349] Starting recovery of all containers
Sep 15 13:38:20.846024 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:20.845988    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 13:38:20.846118 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:20.846097    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 13:38:20.849481 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.849454    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kx98l"
Sep 15 13:38:20.855938 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.855806    2582 manager.go:354] Recovery completed
Sep 15 13:38:20.856562 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.856540    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kx98l"
Sep 15 13:38:20.857343 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:20.857320    2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 15 13:38:20.860680 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.860667    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 13:38:20.862428 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.862411    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 13:38:20.862511 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.862441    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 13:38:20.862511 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.862454    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 13:38:20.863239 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.863223    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 13:38:20.863239 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.863237    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 13:38:20.863407 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.863255    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 13:38:20.863407 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.863238    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 13:38:20.864046 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.864033    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 13:38:20.864102 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.864070    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 13:38:20.864102 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.864085    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 13:38:20.864445 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:20.864379    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-252.ec2.internal.18d5821ffdbf8e57  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-252.ec2.internal,UID:ip-10-0-138-252.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-252.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-252.ec2.internal,},FirstTimestamp:2026-09-15 13:38:20.862426711 +0000 UTC m=+0.395459161,LastTimestamp:2026-09-15 13:38:20.862426711 +0000 UTC m=+0.395459161,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-252.ec2.internal,}"
Sep 15 13:38:20.865293 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.865279    2582 policy_none.go:49] "None policy: Start"
Sep 15 13:38:20.865356 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.865298    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 13:38:20.865356 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.865312    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 13:38:20.867327 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.867313    2582 policy_none.go:47] "Start"
Sep 15 13:38:20.895718 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.895699    2582 manager.go:337] "Starting Device Plugin manager"
Sep 15 13:38:20.895809 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:20.895740    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 13:38:20.895809 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.895751    2582 server.go:87] "Starting device plugin registration server"
Sep 15 13:38:20.896144 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.896123    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 13:38:20.896470 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.896144    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 13:38:20.896470 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.896224    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 13:38:20.896470 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.896307    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 13:38:20.896470 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.896317    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 13:38:20.896984 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:20.896961    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 13:38:20.897093 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:20.897041    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:20.979743 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.979703    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 13:38:20.981807 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.981053    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 13:38:20.981807 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.981101    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 13:38:20.981807 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.981132    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 13:38:20.981807 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:20.981237    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 13:38:20.983250 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.983231    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 13:38:20.996903 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.996884    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 13:38:20.997762 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.997745    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 13:38:20.997841 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.997781    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 13:38:20.997841 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.997797    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 13:38:20.997841 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:20.997828    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.005864 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.005804    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.005864 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:21.005827    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-252.ec2.internal\": node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:21.036643 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:21.036612    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:21.082345 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.082293    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"]
Sep 15 13:38:21.082429 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.082391    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 13:38:21.084124 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.084109    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 13:38:21.084207 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.084137    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 13:38:21.084207 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.084148    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 13:38:21.086380 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.086367    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 13:38:21.086520 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.086506    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.086560 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.086537    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 13:38:21.087129 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.087114    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 13:38:21.087191 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.087145    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 13:38:21.087191 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.087156    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 13:38:21.087255 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.087116    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 13:38:21.087255 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.087213    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 13:38:21.087255 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.087223    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 13:38:21.089318 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.089305    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.089368 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.089330    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 13:38:21.090057 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.090041    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 13:38:21.090159 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.090065    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 13:38:21.090159 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.090074    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 13:38:21.113825 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:21.113802    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-252.ec2.internal\" not found" node="ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.118356 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:21.118338    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-252.ec2.internal\" not found" node="ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.136906 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:21.136886    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:21.145510 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.145491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76b83e804e1c7d5c53a2939b9adbecc1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"76b83e804e1c7d5c53a2939b9adbecc1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.145585 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.145532    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76b83e804e1c7d5c53a2939b9adbecc1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"76b83e804e1c7d5c53a2939b9adbecc1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.145585 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.145549    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd3376ef08c7e4621c42e45696eb6b8e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-252.ec2.internal\" (UID: \"cd3376ef08c7e4621c42e45696eb6b8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.237816 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:21.237775    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:21.246144 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.246125    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd3376ef08c7e4621c42e45696eb6b8e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-252.ec2.internal\" (UID: \"cd3376ef08c7e4621c42e45696eb6b8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.246206 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.246152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76b83e804e1c7d5c53a2939b9adbecc1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"76b83e804e1c7d5c53a2939b9adbecc1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.246206 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.246170    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76b83e804e1c7d5c53a2939b9adbecc1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"76b83e804e1c7d5c53a2939b9adbecc1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.246281 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.246238    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd3376ef08c7e4621c42e45696eb6b8e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-252.ec2.internal\" (UID: \"cd3376ef08c7e4621c42e45696eb6b8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.246321 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.246242    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76b83e804e1c7d5c53a2939b9adbecc1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"76b83e804e1c7d5c53a2939b9adbecc1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.246321 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:21.246244    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76b83e804e1c7d5c53a2939b9adbecc1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"76b83e804e1c7d5c53a2939b9adbecc1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:21.338625 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:21.338561    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:23.072261 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.072234    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:23.078233 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.077587    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:23.093797 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:23.093775    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:23.194358 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:23.194315    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:23.294890 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:23.294809    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:23.395393 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:23.395364    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:23.414161 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.414119    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 13:38:23.427921 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.427892    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 13:38:23.428051 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.428032    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 13:38:23.428096 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.428063    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 13:38:23.492849 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.492821    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 13:38:23.495714 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:23.495698    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:23.517900 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.517862    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 13:33:22 +0000 UTC" deadline="2028-02-12 17:29:20.720840744 +0000 UTC"
Sep 15 13:38:23.517900 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.517891    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12363h50m57.202952841s"
Sep 15 13:38:23.596254 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:23.596183    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:23.696686 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:23.696648    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 15 13:38:23.795523 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.795491    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 13:38:23.799360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.799339    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:23.817213 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.817193    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 13:38:23.818118 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.818099    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"
Sep 15 13:38:23.818931 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:23.818908    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3376ef08c7e4621c42e45696eb6b8e.slice/crio-ac4ba27a5f379349f9610d6df593a6ba7e73bd449426f6d86889962f019bc04c WatchSource:0}: Error finding container ac4ba27a5f379349f9610d6df593a6ba7e73bd449426f6d86889962f019bc04c: Status 404 returned error can't find the container with id ac4ba27a5f379349f9610d6df593a6ba7e73bd449426f6d86889962f019bc04c
Sep 15 13:38:23.822403 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.822388    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 13:38:23.827692 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:23.827674    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 13:38:24.473642 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.473610    2582 apiserver.go:52] "Watching apiserver"
Sep 15 13:38:24.483525 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.483487    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 13:38:24.484731 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.484700    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-s2l72","openshift-network-diagnostics/network-check-target-j5pmd","openshift-network-operator/iptables-alerter-zlxcv","kube-system/global-pull-secret-syncer-fbj84","kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal","openshift-cluster-node-tuning-operator/tuned-7c5tp","openshift-image-registry/node-ca-b7tq6","openshift-multus/multus-additional-cni-plugins-dxgrp","openshift-multus/network-metrics-daemon-5kxxr","openshift-ovn-kubernetes/ovnkube-node-ttnsl","kube-system/konnectivity-agent-92nmn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt","openshift-dns/node-resolver-lhd7g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"]
Sep 15 13:38:24.487331 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.487305    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-92nmn"
Sep 15 13:38:24.489477 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.489453    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:24.489589 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:24.489535    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:24.489775 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.489757    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 13:38:24.489846 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.489800    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 13:38:24.489901 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.489848    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bdv7x\""
Sep 15 13:38:24.491648 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.491629    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zlxcv"
Sep 15 13:38:24.493989 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.493685    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fqhnn\""
Sep 15 13:38:24.493989 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.493694    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 13:38:24.493989 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.493714    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 13:38:24.493989 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.493923    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.494275 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.493991    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 13:38:24.496068 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.495946    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 13:38:24.496068 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.496054    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 13:38:24.496289 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.496228    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lhd7g"
Sep 15 13:38:24.496764 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.496748    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 13:38:24.496764 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.496755    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f7cmq\""
Sep 15 13:38:24.497030 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.497001    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 13:38:24.498444 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.498279    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 13:38:24.498444 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.498339    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 13:38:24.498444 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.498348    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.498444 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.498354    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9spgr\""
Sep 15 13:38:24.499110 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.499096    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 13:38:24.500561 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.500496    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 13:38:24.500561 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.500543    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v86dx\""
Sep 15 13:38:24.500716 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.500545    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 13:38:24.501695 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.501665    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.504081 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.504060    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 13:38:24.504193 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.504065    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cbz9x\""
Sep 15 13:38:24.504356 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.504338    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.504475 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.504362    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 13:38:24.504475 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.504343    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 13:38:24.506377 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.506360    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:24.506469 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.506411    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 13:38:24.506469 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:24.506435    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:24.506682 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.506666    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 13:38:24.506743 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.506719    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kxwkx\""
Sep 15 13:38:24.508905 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.508887    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.509462 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509440    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbdmc\" (UniqueName: \"kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc\") pod \"network-check-target-j5pmd\" (UID: \"ef84ca8e-d344-4d38-b015-8a76a03e5dfa\") " pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:24.509561 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1476f2-df89-474d-bbce-2258e2487080-tmp-dir\") pod \"node-resolver-lhd7g\" (UID: \"9a1476f2-df89-474d-bbce-2258e2487080\") " pod="openshift-dns/node-resolver-lhd7g"
Sep 15 13:38:24.509561 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509497    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-socket-dir\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.509561 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509520    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-etc-kubernetes\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.509561 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srqnb\" (UniqueName: \"kubernetes.io/projected/9a1476f2-df89-474d-bbce-2258e2487080-kube-api-access-srqnb\") pod \"node-resolver-lhd7g\" (UID: \"9a1476f2-df89-474d-bbce-2258e2487080\") " pod="openshift-dns/node-resolver-lhd7g"
Sep 15 13:38:24.509754 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509571    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-cnibin\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.509754 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-multus-socket-dir-parent\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.509754 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-kubernetes\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.509754 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509662    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-sys\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.509754 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-tuned\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.509754 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509712    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103-host-slash\") pod \"iptables-alerter-zlxcv\" (UID: \"fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103\") " pod="openshift-network-operator/iptables-alerter-zlxcv"
Sep 15 13:38:24.509754 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509752    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mls8\" (UniqueName: \"kubernetes.io/projected/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-kube-api-access-8mls8\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.510079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509775    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-multus-cni-dir\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.510079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-hostroot\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.510079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-lib-modules\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.510079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509820    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-host\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.510079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hv96\" (UniqueName: \"kubernetes.io/projected/fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103-kube-api-access-2hv96\") pod \"iptables-alerter-zlxcv\" (UID: \"fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103\") " pod="openshift-network-operator/iptables-alerter-zlxcv"
Sep 15 13:38:24.510079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a1476f2-df89-474d-bbce-2258e2487080-hosts-file\") pod \"node-resolver-lhd7g\" (UID: \"9a1476f2-df89-474d-bbce-2258e2487080\") " pod="openshift-dns/node-resolver-lhd7g"
Sep 15 13:38:24.510079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509912    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-device-dir\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.510079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac943d06-bf47-4dbf-8d40-8c63e0c48007-konnectivity-ca\") pod \"konnectivity-agent-92nmn\" (UID: \"ac943d06-bf47-4dbf-8d40-8c63e0c48007\") " pod="kube-system/konnectivity-agent-92nmn"
Sep 15 13:38:24.510079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.509985    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae90d50d-e61a-43d9-8286-dc01dc000417-cni-binary-copy\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.510079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-var-lib-cni-bin\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.510079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-var-lib-cni-multus\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.510079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.510079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510090    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac943d06-bf47-4dbf-8d40-8c63e0c48007-agent-certs\") pod \"konnectivity-agent-92nmn\" (UID: \"ac943d06-bf47-4dbf-8d40-8c63e0c48007\") " pod="kube-system/konnectivity-agent-92nmn"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510113    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-var-lib-kubelet\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-run-multus-certs\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwz25\" (UniqueName: \"kubernetes.io/projected/ae90d50d-e61a-43d9-8286-dc01dc000417-kube-api-access-vwz25\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-sysctl-conf\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-systemd\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfw9c\" (UniqueName: \"kubernetes.io/projected/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-kube-api-access-zfw9c\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510277    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103-iptables-alerter-script\") pod \"iptables-alerter-zlxcv\" (UID: \"fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103\") " pod="openshift-network-operator/iptables-alerter-zlxcv"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510314    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510336    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-system-cni-dir\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510358    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-run-k8s-cni-cncf-io\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-run-netns\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-run\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-registration-dir\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-sys-fs\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-sysconfig\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.510842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510518    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-sysctl-d\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.511625 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-var-lib-kubelet\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.511625 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-multus-conf-dir\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.511625 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae90d50d-e61a-43d9-8286-dc01dc000417-multus-daemon-config\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.511625 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510613    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-modprobe-d\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.511625 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.510636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-tmp\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.511625 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.511073    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 13:38:24.511625 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.511124    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 13:38:24.511625 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.511135    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7v9md\""
Sep 15 13:38:24.511625 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.511376    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:24.511625 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:24.511437    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:24.512091 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.511925    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 13:38:24.512091 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.511955    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 13:38:24.512091 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.511967    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 13:38:24.512091 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.511971    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 13:38:24.512275 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.512228    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 13:38:24.513792 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.513656    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b7tq6"
Sep 15 13:38:24.515454 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.515437    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 13:38:24.515724 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.515702    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-twdbz\""
Sep 15 13:38:24.515914 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.515900    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 13:38:24.516114 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.516100    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 13:38:24.531615 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.531590    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kpsqw"
Sep 15 13:38:24.541813 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.541788    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kpsqw"
Sep 15 13:38:24.600612 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.600583    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 13:38:24.610829 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.610801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-socket-dir\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.610973 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.610833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-multus-socket-dir-parent\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.610973 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.610861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-kubernetes\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.610973 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.610881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-tuned\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.610973 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.610899    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-slash\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.610973 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.610922    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvm9\" (UniqueName: \"kubernetes.io/projected/7edf22ea-7f60-4d71-95db-8594c103b405-kube-api-access-qcvm9\") pod \"node-ca-b7tq6\" (UID: \"7edf22ea-7f60-4d71-95db-8594c103b405\") " pod="openshift-image-registry/node-ca-b7tq6"
Sep 15 13:38:24.610973 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.610932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-multus-socket-dir-parent\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.610973 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.610939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mls8\" (UniqueName: \"kubernetes.io/projected/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-kube-api-access-8mls8\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.610973 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.610972    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-socket-dir\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.610979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-kubernetes\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.610986    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-kubelet\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-run-openvswitch\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-systemd\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611072    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611108    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac943d06-bf47-4dbf-8d40-8c63e0c48007-konnectivity-ca\") pod \"konnectivity-agent-92nmn\" (UID: \"ac943d06-bf47-4dbf-8d40-8c63e0c48007\") " pod="kube-system/konnectivity-agent-92nmn"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611121    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-systemd\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611158    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-systemd-units\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611298    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-node-log\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611321    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/464b7bb4-eaaa-4752-9055-3284a71d4e48-ovnkube-config\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611327    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/464b7bb4-eaaa-4752-9055-3284a71d4e48-ovn-node-metrics-cert\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.611364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-system-cni-dir\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611527    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-run-k8s-cni-cncf-io\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611615    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-cni-bin\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-dbus\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611628    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-system-cni-dir\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611668    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac943d06-bf47-4dbf-8d40-8c63e0c48007-konnectivity-ca\") pod \"konnectivity-agent-92nmn\" (UID: \"ac943d06-bf47-4dbf-8d40-8c63e0c48007\") " pod="kube-system/konnectivity-agent-92nmn"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611677    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7edf22ea-7f60-4d71-95db-8594c103b405-host\") pod \"node-ca-b7tq6\" (UID: \"7edf22ea-7f60-4d71-95db-8594c103b405\") " pod="openshift-image-registry/node-ca-b7tq6"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611691    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-run-k8s-cni-cncf-io\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611725    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-registration-dir\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611770    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-registration-dir\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-sysctl-d\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-var-lib-kubelet\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae90d50d-e61a-43d9-8286-dc01dc000417-multus-daemon-config\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611850    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-modprobe-d\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.612038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611872    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-tmp\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611881    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-var-lib-kubelet\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611899    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-etc-openvswitch\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611960    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-modprobe-d\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-kubelet-config\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611898    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-sysctl-d\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.611993    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdmc\" (UniqueName: \"kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc\") pod \"network-check-target-j5pmd\" (UID: \"ef84ca8e-d344-4d38-b015-8a76a03e5dfa\") " pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1476f2-df89-474d-bbce-2258e2487080-tmp-dir\") pod \"node-resolver-lhd7g\" (UID: \"9a1476f2-df89-474d-bbce-2258e2487080\") " pod="openshift-dns/node-resolver-lhd7g"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-etc-kubernetes\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-run-netns\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612109    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srqnb\" (UniqueName: \"kubernetes.io/projected/9a1476f2-df89-474d-bbce-2258e2487080-kube-api-access-srqnb\") pod \"node-resolver-lhd7g\" (UID: \"9a1476f2-df89-474d-bbce-2258e2487080\") " pod="openshift-dns/node-resolver-lhd7g"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-cnibin\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612170    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-sys\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612222    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-var-lib-openvswitch\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612245    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-run-ovn\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-run-ovn-kubernetes\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.612699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612295    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs\") pod \"network-metrics-daemon-5kxxr\" (UID: \"0258dc54-5388-49fa-bd24-0ee885930283\") " pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103-host-slash\") pod \"iptables-alerter-zlxcv\" (UID: \"fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103\") " pod="openshift-network-operator/iptables-alerter-zlxcv"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-multus-cni-dir\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612362    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1476f2-df89-474d-bbce-2258e2487080-tmp-dir\") pod \"node-resolver-lhd7g\" (UID: \"9a1476f2-df89-474d-bbce-2258e2487080\") " pod="openshift-dns/node-resolver-lhd7g"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612432    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-hostroot\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-cnibin\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612447    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-etc-kubernetes\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae90d50d-e61a-43d9-8286-dc01dc000417-multus-daemon-config\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103-host-slash\") pod \"iptables-alerter-zlxcv\" (UID: \"fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103\") " pod="openshift-network-operator/iptables-alerter-zlxcv"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612367    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-hostroot\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612546    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-multus-cni-dir\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-lib-modules\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-host\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612613    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-sys\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/464b7bb4-eaaa-4752-9055-3284a71d4e48-env-overrides\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612700    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-lib-modules\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612733    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-var-lib-kubelet\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.613473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-cni-binary-copy\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612775    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hv96\" (UniqueName: \"kubernetes.io/projected/fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103-kube-api-access-2hv96\") pod \"iptables-alerter-zlxcv\" (UID: \"fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103\") " pod="openshift-network-operator/iptables-alerter-zlxcv"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612785    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-host\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a1476f2-df89-474d-bbce-2258e2487080-hosts-file\") pod \"node-resolver-lhd7g\" (UID: \"9a1476f2-df89-474d-bbce-2258e2487080\") " pod="openshift-dns/node-resolver-lhd7g"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612842    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-var-lib-kubelet\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-device-dir\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a1476f2-df89-474d-bbce-2258e2487080-hosts-file\") pod \"node-resolver-lhd7g\" (UID: \"9a1476f2-df89-474d-bbce-2258e2487080\") " pod="openshift-dns/node-resolver-lhd7g"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.612969    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae90d50d-e61a-43d9-8286-dc01dc000417-cni-binary-copy\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-var-lib-cni-bin\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613033    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-device-dir\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-var-lib-cni-multus\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac943d06-bf47-4dbf-8d40-8c63e0c48007-agent-certs\") pod \"konnectivity-agent-92nmn\" (UID: \"ac943d06-bf47-4dbf-8d40-8c63e0c48007\") " pod="kube-system/konnectivity-agent-92nmn"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-var-lib-cni-multus\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-run-multus-certs\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613273    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwz25\" (UniqueName: \"kubernetes.io/projected/ae90d50d-e61a-43d9-8286-dc01dc000417-kube-api-access-vwz25\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613300    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-sysctl-conf\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfw9c\" (UniqueName: \"kubernetes.io/projected/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-kube-api-access-zfw9c\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.614360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613373    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-cni-netd\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/464b7bb4-eaaa-4752-9055-3284a71d4e48-ovnkube-script-lib\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613430    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r7c7\" (UniqueName: \"kubernetes.io/projected/0258dc54-5388-49fa-bd24-0ee885930283-kube-api-access-6r7c7\") pod \"network-metrics-daemon-5kxxr\" (UID: \"0258dc54-5388-49fa-bd24-0ee885930283\") " pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103-iptables-alerter-script\") pod \"iptables-alerter-zlxcv\" (UID: \"fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103\") " pod="openshift-network-operator/iptables-alerter-zlxcv"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613484    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-run-netns\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-run\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-var-lib-cni-bin\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613557    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-cnibin\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613563    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae90d50d-e61a-43d9-8286-dc01dc000417-cni-binary-copy\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-run-systemd\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-run-netns\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7edf22ea-7f60-4d71-95db-8594c103b405-serviceca\") pod \"node-ca-b7tq6\" (UID: \"7edf22ea-7f60-4d71-95db-8594c103b405\") " pod="openshift-image-registry/node-ca-b7tq6"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613621    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-run\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613628    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-sysctl-conf\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-sys-fs\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613664    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-host-run-multus-certs\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-sysconfig\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.615121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-system-cni-dir\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.615779 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.615779 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613755    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-sys-fs\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.615779 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr8h8\" (UniqueName: \"kubernetes.io/projected/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-kube-api-access-wr8h8\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.615779 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-sysconfig\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.615779 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppfn8\" (UniqueName: \"kubernetes.io/projected/464b7bb4-eaaa-4752-9055-3284a71d4e48-kube-api-access-ppfn8\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.615779 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-multus-conf-dir\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.615779 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613903    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae90d50d-e61a-43d9-8286-dc01dc000417-multus-conf-dir\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.615779 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.613899    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-log-socket\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.615779 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.614068    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103-iptables-alerter-script\") pod \"iptables-alerter-zlxcv\" (UID: \"fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103\") " pod="openshift-network-operator/iptables-alerter-zlxcv"
Sep 15 13:38:24.615779 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.615307    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-tmp\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.615779 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.615456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-etc-tuned\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.616365 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.616349    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac943d06-bf47-4dbf-8d40-8c63e0c48007-agent-certs\") pod \"konnectivity-agent-92nmn\" (UID: \"ac943d06-bf47-4dbf-8d40-8c63e0c48007\") " pod="kube-system/konnectivity-agent-92nmn"
Sep 15 13:38:24.618236 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:24.618217    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 13:38:24.618236 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:24.618239    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 13:38:24.618358 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:24.618249    2582 projected.go:196] Error preparing data for projected volume kube-api-access-sbdmc for pod openshift-network-diagnostics/network-check-target-j5pmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:24.618358 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:24.618299    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc podName:ef84ca8e-d344-4d38-b015-8a76a03e5dfa nodeName:}" failed. No retries permitted until 2026-09-15 13:38:25.118284846 +0000 UTC m=+2.996187642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sbdmc" (UniqueName: "kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc") pod "network-check-target-j5pmd" (UID: "ef84ca8e-d344-4d38-b015-8a76a03e5dfa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:24.620547 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.620500    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srqnb\" (UniqueName: \"kubernetes.io/projected/9a1476f2-df89-474d-bbce-2258e2487080-kube-api-access-srqnb\") pod \"node-resolver-lhd7g\" (UID: \"9a1476f2-df89-474d-bbce-2258e2487080\") " pod="openshift-dns/node-resolver-lhd7g"
Sep 15 13:38:24.621104 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.621080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mls8\" (UniqueName: \"kubernetes.io/projected/9b3b9786-6a4f-4c44-8eec-6602a1acd59d-kube-api-access-8mls8\") pod \"aws-ebs-csi-driver-node-xqtxt\" (UID: \"9b3b9786-6a4f-4c44-8eec-6602a1acd59d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.621641 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.621401    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfw9c\" (UniqueName: \"kubernetes.io/projected/5ed7f5fd-fd66-4f31-a0e8-f6453fd55829-kube-api-access-zfw9c\") pod \"tuned-7c5tp\" (UID: \"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829\") " pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.621641 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.621554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwz25\" (UniqueName: \"kubernetes.io/projected/ae90d50d-e61a-43d9-8286-dc01dc000417-kube-api-access-vwz25\") pod \"multus-s2l72\" (UID: \"ae90d50d-e61a-43d9-8286-dc01dc000417\") " pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.621748 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.621656    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hv96\" (UniqueName: \"kubernetes.io/projected/fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103-kube-api-access-2hv96\") pod \"iptables-alerter-zlxcv\" (UID: \"fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103\") " pod="openshift-network-operator/iptables-alerter-zlxcv"
Sep 15 13:38:24.641423 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.641382    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" event={"ID":"cd3376ef08c7e4621c42e45696eb6b8e","Type":"ContainerStarted","Data":"ac4ba27a5f379349f9610d6df593a6ba7e73bd449426f6d86889962f019bc04c"}
Sep 15 13:38:24.674191 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:24.674160    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b83e804e1c7d5c53a2939b9adbecc1.slice/crio-987eeead9680af21e128f12e4863a4fa4003357257ee4d047d649d918fe9d261 WatchSource:0}: Error finding container 987eeead9680af21e128f12e4863a4fa4003357257ee4d047d649d918fe9d261: Status 404 returned error can't find the container with id 987eeead9680af21e128f12e4863a4fa4003357257ee4d047d649d918fe9d261
Sep 15 13:38:24.714585 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714557    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-cni-binary-copy\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.714811 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-cni-netd\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.714811 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714624    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/464b7bb4-eaaa-4752-9055-3284a71d4e48-ovnkube-script-lib\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.714811 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714646    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7c7\" (UniqueName: \"kubernetes.io/projected/0258dc54-5388-49fa-bd24-0ee885930283-kube-api-access-6r7c7\") pod \"network-metrics-daemon-5kxxr\" (UID: \"0258dc54-5388-49fa-bd24-0ee885930283\") " pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:24.714811 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714670    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-cnibin\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.714811 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-run-systemd\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.714811 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714689    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-cni-netd\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.714811 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7edf22ea-7f60-4d71-95db-8594c103b405-serviceca\") pod \"node-ca-b7tq6\" (UID: \"7edf22ea-7f60-4d71-95db-8594c103b405\") " pod="openshift-image-registry/node-ca-b7tq6"
Sep 15 13:38:24.714811 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-system-cni-dir\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.714811 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.714811 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr8h8\" (UniqueName: \"kubernetes.io/projected/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-kube-api-access-wr8h8\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.714811 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-run-systemd\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppfn8\" (UniqueName: \"kubernetes.io/projected/464b7bb4-eaaa-4752-9055-3284a71d4e48-kube-api-access-ppfn8\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-log-socket\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-slash\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714900    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvm9\" (UniqueName: \"kubernetes.io/projected/7edf22ea-7f60-4d71-95db-8594c103b405-kube-api-access-qcvm9\") pod \"node-ca-b7tq6\" (UID: \"7edf22ea-7f60-4d71-95db-8594c103b405\") " pod="openshift-image-registry/node-ca-b7tq6"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-kubelet\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-run-openvswitch\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.714996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-systemd-units\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-node-log\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/464b7bb4-eaaa-4752-9055-3284a71d4e48-ovnkube-config\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/464b7bb4-eaaa-4752-9055-3284a71d4e48-ovn-node-metrics-cert\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715093    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-cni-bin\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715130    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-cni-binary-copy\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715204    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-system-cni-dir\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715242    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-node-log\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.715346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715295    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-kubelet\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715331    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-run-openvswitch\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715346    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715377    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-systemd-units\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715404    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-slash\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715431    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-cnibin\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-dbus\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7edf22ea-7f60-4d71-95db-8594c103b405-host\") pod \"node-ca-b7tq6\" (UID: \"7edf22ea-7f60-4d71-95db-8594c103b405\") " pod="openshift-image-registry/node-ca-b7tq6"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-etc-openvswitch\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7edf22ea-7f60-4d71-95db-8594c103b405-serviceca\") pod \"node-ca-b7tq6\" (UID: \"7edf22ea-7f60-4d71-95db-8594c103b405\") " pod="openshift-image-registry/node-ca-b7tq6"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-kubelet-config\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-run-netns\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-log-socket\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-var-lib-openvswitch\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-run-ovn\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-run-ovn-kubernetes\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715599    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715606    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/464b7bb4-eaaa-4752-9055-3284a71d4e48-ovnkube-script-lib\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715641    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7edf22ea-7f60-4d71-95db-8594c103b405-host\") pod \"node-ca-b7tq6\" (UID: \"7edf22ea-7f60-4d71-95db-8594c103b405\") " pod="openshift-image-registry/node-ca-b7tq6"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715655    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-kubelet-config\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715679    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-run-netns\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715679    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-etc-openvswitch\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715694    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-var-lib-openvswitch\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs\") pod \"network-metrics-daemon-5kxxr\" (UID: \"0258dc54-5388-49fa-bd24-0ee885930283\") " pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:24.715695    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715727    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-run-ovn\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715752    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-run-ovn-kubernetes\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/464b7bb4-eaaa-4752-9055-3284a71d4e48-env-overrides\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:24.715780    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs podName:0258dc54-5388-49fa-bd24-0ee885930283 nodeName:}" failed. No retries permitted until 2026-09-15 13:38:25.215767628 +0000 UTC m=+3.093670425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs") pod "network-metrics-daemon-5kxxr" (UID: "0258dc54-5388-49fa-bd24-0ee885930283") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715792    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/464b7bb4-eaaa-4752-9055-3284a71d4e48-ovnkube-config\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715801    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/464b7bb4-eaaa-4752-9055-3284a71d4e48-host-cni-bin\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.715855    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-dbus\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:24.716708 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:24.715891    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:24.717284 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:24.715924    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret podName:b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe nodeName:}" failed. No retries permitted until 2026-09-15 13:38:25.215913692 +0000 UTC m=+3.093816493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret") pod "global-pull-secret-syncer-fbj84" (UID: "b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe") : object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:24.717284 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.716210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.717284 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.716282    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/464b7bb4-eaaa-4752-9055-3284a71d4e48-env-overrides\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.718377 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.718360    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/464b7bb4-eaaa-4752-9055-3284a71d4e48-ovn-node-metrics-cert\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.724522 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.724467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppfn8\" (UniqueName: \"kubernetes.io/projected/464b7bb4-eaaa-4752-9055-3284a71d4e48-kube-api-access-ppfn8\") pod \"ovnkube-node-ttnsl\" (UID: \"464b7bb4-eaaa-4752-9055-3284a71d4e48\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.725561 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.725542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvm9\" (UniqueName: \"kubernetes.io/projected/7edf22ea-7f60-4d71-95db-8594c103b405-kube-api-access-qcvm9\") pod \"node-ca-b7tq6\" (UID: \"7edf22ea-7f60-4d71-95db-8594c103b405\") " pod="openshift-image-registry/node-ca-b7tq6"
Sep 15 13:38:24.726472 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.726450    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr8h8\" (UniqueName: \"kubernetes.io/projected/e42c6f90-64b3-4a55-bf82-c9ee9a126dc9-kube-api-access-wr8h8\") pod \"multus-additional-cni-plugins-dxgrp\" (UID: \"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9\") " pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.726864 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.726843    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r7c7\" (UniqueName: \"kubernetes.io/projected/0258dc54-5388-49fa-bd24-0ee885930283-kube-api-access-6r7c7\") pod \"network-metrics-daemon-5kxxr\" (UID: \"0258dc54-5388-49fa-bd24-0ee885930283\") " pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:24.799280 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.799246    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-92nmn"
Sep 15 13:38:24.808493 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.808465    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zlxcv"
Sep 15 13:38:24.820309 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.820279    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s2l72"
Sep 15 13:38:24.825271 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.825248    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lhd7g"
Sep 15 13:38:24.828414 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.828395    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 13:38:24.838439 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.838416    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7c5tp"
Sep 15 13:38:24.846200 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.846179    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt"
Sep 15 13:38:24.853882 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.853864    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dxgrp"
Sep 15 13:38:24.861512 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.861491    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:24.868327 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:24.868301    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b7tq6"
Sep 15 13:38:25.118419 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.118328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdmc\" (UniqueName: \"kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc\") pod \"network-check-target-j5pmd\" (UID: \"ef84ca8e-d344-4d38-b015-8a76a03e5dfa\") " pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:25.118571 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:25.118512    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 13:38:25.118571 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:25.118540    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 13:38:25.118571 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:25.118554    2582 projected.go:196] Error preparing data for projected volume kube-api-access-sbdmc for pod openshift-network-diagnostics/network-check-target-j5pmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:25.118724 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:25.118618    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc podName:ef84ca8e-d344-4d38-b015-8a76a03e5dfa nodeName:}" failed. No retries permitted until 2026-09-15 13:38:26.118599843 +0000 UTC m=+3.996502639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbdmc" (UniqueName: "kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc") pod "network-check-target-j5pmd" (UID: "ef84ca8e-d344-4d38-b015-8a76a03e5dfa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:25.219342 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.219300    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs\") pod \"network-metrics-daemon-5kxxr\" (UID: \"0258dc54-5388-49fa-bd24-0ee885930283\") " pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:25.219527 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.219349    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:25.219527 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:25.219456    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:25.219527 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:25.219458    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:25.219527 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:25.219515    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret podName:b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe nodeName:}" failed. No retries permitted until 2026-09-15 13:38:26.219499641 +0000 UTC m=+4.097402440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret") pod "global-pull-secret-syncer-fbj84" (UID: "b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe") : object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:25.219527 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:25.219529    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs podName:0258dc54-5388-49fa-bd24-0ee885930283 nodeName:}" failed. No retries permitted until 2026-09-15 13:38:26.219522924 +0000 UTC m=+4.097425721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs") pod "network-metrics-daemon-5kxxr" (UID: "0258dc54-5388-49fa-bd24-0ee885930283") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:25.352656 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:25.352506    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed7f5fd_fd66_4f31_a0e8_f6453fd55829.slice/crio-1c7286eb55770ba4f7f7a867e919151600786d42624667880b33bbe600ddd8df WatchSource:0}: Error finding container 1c7286eb55770ba4f7f7a867e919151600786d42624667880b33bbe600ddd8df: Status 404 returned error can't find the container with id 1c7286eb55770ba4f7f7a867e919151600786d42624667880b33bbe600ddd8df
Sep 15 13:38:25.353095 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:25.352966    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae90d50d_e61a_43d9_8286_dc01dc000417.slice/crio-be40462104e1c3196bc864a324511de11fd0398a04b0b88848e3be8132096285 WatchSource:0}: Error finding container be40462104e1c3196bc864a324511de11fd0398a04b0b88848e3be8132096285: Status 404 returned error can't find the container with id be40462104e1c3196bc864a324511de11fd0398a04b0b88848e3be8132096285
Sep 15 13:38:25.356092 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:25.356070    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac943d06_bf47_4dbf_8d40_8c63e0c48007.slice/crio-515e27ead1777094f3a2465df0c846e7b97c528d042581646eef0bbc7c7f4a49 WatchSource:0}: Error finding container 515e27ead1777094f3a2465df0c846e7b97c528d042581646eef0bbc7c7f4a49: Status 404 returned error can't find the container with id 515e27ead1777094f3a2465df0c846e7b97c528d042581646eef0bbc7c7f4a49
Sep 15 13:38:25.358333 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:25.358268    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3b9786_6a4f_4c44_8eec_6602a1acd59d.slice/crio-69c91aa6d1185c4dc9fd1c23be3cfb6be66c1f9b61591f2e53c43261a48c84da WatchSource:0}: Error finding container 69c91aa6d1185c4dc9fd1c23be3cfb6be66c1f9b61591f2e53c43261a48c84da: Status 404 returned error can't find the container with id 69c91aa6d1185c4dc9fd1c23be3cfb6be66c1f9b61591f2e53c43261a48c84da
Sep 15 13:38:25.358707 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:25.358690    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464b7bb4_eaaa_4752_9055_3284a71d4e48.slice/crio-9b60526a2126e317ab36807044718eda7c3eba3908ef3f2f26ac6a6113882eaa WatchSource:0}: Error finding container 9b60526a2126e317ab36807044718eda7c3eba3908ef3f2f26ac6a6113882eaa: Status 404 returned error can't find the container with id 9b60526a2126e317ab36807044718eda7c3eba3908ef3f2f26ac6a6113882eaa
Sep 15 13:38:25.359554 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:25.359535    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1476f2_df89_474d_bbce_2258e2487080.slice/crio-a42c53f47acea6d418cfa1a3c57c412508099e9a4f85a37702a5deb7b3bfe9f9 WatchSource:0}: Error finding container a42c53f47acea6d418cfa1a3c57c412508099e9a4f85a37702a5deb7b3bfe9f9: Status 404 returned error can't find the container with id a42c53f47acea6d418cfa1a3c57c412508099e9a4f85a37702a5deb7b3bfe9f9
Sep 15 13:38:25.382725 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:25.382705    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae7c57f_7dfa_47ae_b61a_6fc3d8eaf103.slice/crio-67f5fdcbeef36169886e4dfa38a2ba3eb67df4410c81e2a73de4663f8f42d9aa WatchSource:0}: Error finding container 67f5fdcbeef36169886e4dfa38a2ba3eb67df4410c81e2a73de4663f8f42d9aa: Status 404 returned error can't find the container with id 67f5fdcbeef36169886e4dfa38a2ba3eb67df4410c81e2a73de4663f8f42d9aa
Sep 15 13:38:25.383418 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:25.383401    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edf22ea_7f60_4d71_95db_8594c103b405.slice/crio-16d27de78ed5c5071024ec3157812fcbdd5491e0f5476cf6cfc973a3a691677c WatchSource:0}: Error finding container 16d27de78ed5c5071024ec3157812fcbdd5491e0f5476cf6cfc973a3a691677c: Status 404 returned error can't find the container with id 16d27de78ed5c5071024ec3157812fcbdd5491e0f5476cf6cfc973a3a691677c
Sep 15 13:38:25.384748 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:25.384731    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42c6f90_64b3_4a55_bf82_c9ee9a126dc9.slice/crio-d1f7ee3877826848fa5673040267e9a5c17a7166adb3584ff03173a732a272d8 WatchSource:0}: Error finding container d1f7ee3877826848fa5673040267e9a5c17a7166adb3584ff03173a732a272d8: Status 404 returned error can't find the container with id d1f7ee3877826848fa5673040267e9a5c17a7166adb3584ff03173a732a272d8
Sep 15 13:38:25.545600 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.545546    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 13:33:24 +0000 UTC" deadline="2028-04-18 22:11:45.016265666 +0000 UTC"
Sep 15 13:38:25.545600 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.545591    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13952h33m19.470677629s"
Sep 15 13:38:25.644889 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.644809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxgrp" event={"ID":"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9","Type":"ContainerStarted","Data":"d1f7ee3877826848fa5673040267e9a5c17a7166adb3584ff03173a732a272d8"}
Sep 15 13:38:25.645902 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.645878    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b7tq6" event={"ID":"7edf22ea-7f60-4d71-95db-8594c103b405","Type":"ContainerStarted","Data":"16d27de78ed5c5071024ec3157812fcbdd5491e0f5476cf6cfc973a3a691677c"}
Sep 15 13:38:25.646990 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.646967    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lhd7g" event={"ID":"9a1476f2-df89-474d-bbce-2258e2487080","Type":"ContainerStarted","Data":"a42c53f47acea6d418cfa1a3c57c412508099e9a4f85a37702a5deb7b3bfe9f9"}
Sep 15 13:38:25.648025 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.647994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl" event={"ID":"464b7bb4-eaaa-4752-9055-3284a71d4e48","Type":"ContainerStarted","Data":"9b60526a2126e317ab36807044718eda7c3eba3908ef3f2f26ac6a6113882eaa"}
Sep 15 13:38:25.649052 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.649032    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-92nmn" event={"ID":"ac943d06-bf47-4dbf-8d40-8c63e0c48007","Type":"ContainerStarted","Data":"515e27ead1777094f3a2465df0c846e7b97c528d042581646eef0bbc7c7f4a49"}
Sep 15 13:38:25.650099 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.650080    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s2l72" event={"ID":"ae90d50d-e61a-43d9-8286-dc01dc000417","Type":"ContainerStarted","Data":"be40462104e1c3196bc864a324511de11fd0398a04b0b88848e3be8132096285"}
Sep 15 13:38:25.651107 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.651089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7c5tp" event={"ID":"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829","Type":"ContainerStarted","Data":"1c7286eb55770ba4f7f7a867e919151600786d42624667880b33bbe600ddd8df"}
Sep 15 13:38:25.652812 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.652793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" event={"ID":"cd3376ef08c7e4621c42e45696eb6b8e","Type":"ContainerStarted","Data":"a5f632a1ed7abce1ebe74c4e83478fc18fa7ef58a7e58e6a00590e9340b3a437"}
Sep 15 13:38:25.653800 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.653778    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zlxcv" event={"ID":"fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103","Type":"ContainerStarted","Data":"67f5fdcbeef36169886e4dfa38a2ba3eb67df4410c81e2a73de4663f8f42d9aa"}
Sep 15 13:38:25.654671 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.654652    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt" event={"ID":"9b3b9786-6a4f-4c44-8eec-6602a1acd59d","Type":"ContainerStarted","Data":"69c91aa6d1185c4dc9fd1c23be3cfb6be66c1f9b61591f2e53c43261a48c84da"}
Sep 15 13:38:25.655731 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.655714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" event={"ID":"76b83e804e1c7d5c53a2939b9adbecc1","Type":"ContainerStarted","Data":"987eeead9680af21e128f12e4863a4fa4003357257ee4d047d649d918fe9d261"}
Sep 15 13:38:25.667632 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:25.667592    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" podStartSLOduration=2.6675780810000003 podStartE2EDuration="2.667578081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:38:25.667266372 +0000 UTC m=+3.545169191" watchObservedRunningTime="2026-09-15 13:38:25.667578081 +0000 UTC m=+3.545480902"
Sep 15 13:38:26.126474 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:26.126394    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdmc\" (UniqueName: \"kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc\") pod \"network-check-target-j5pmd\" (UID: \"ef84ca8e-d344-4d38-b015-8a76a03e5dfa\") " pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:26.126648 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:26.126630    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 13:38:26.126707 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:26.126661    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 13:38:26.126707 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:26.126674    2582 projected.go:196] Error preparing data for projected volume kube-api-access-sbdmc for pod openshift-network-diagnostics/network-check-target-j5pmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:26.126811 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:26.126729    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc podName:ef84ca8e-d344-4d38-b015-8a76a03e5dfa nodeName:}" failed. No retries permitted until 2026-09-15 13:38:28.126712855 +0000 UTC m=+6.004615668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbdmc" (UniqueName: "kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc") pod "network-check-target-j5pmd" (UID: "ef84ca8e-d344-4d38-b015-8a76a03e5dfa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:26.226882 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:26.226843    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs\") pod \"network-metrics-daemon-5kxxr\" (UID: \"0258dc54-5388-49fa-bd24-0ee885930283\") " pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:26.227057 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:26.227038    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:26.227201 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:26.227183    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:26.227276 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:26.227251    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret podName:b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe nodeName:}" failed. No retries permitted until 2026-09-15 13:38:28.227233372 +0000 UTC m=+6.105136186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret") pod "global-pull-secret-syncer-fbj84" (UID: "b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe") : object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:26.227641 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:26.227623    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:26.227725 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:26.227676    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs podName:0258dc54-5388-49fa-bd24-0ee885930283 nodeName:}" failed. No retries permitted until 2026-09-15 13:38:28.227660238 +0000 UTC m=+6.105563040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs") pod "network-metrics-daemon-5kxxr" (UID: "0258dc54-5388-49fa-bd24-0ee885930283") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:26.395449 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:26.395371    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 13:38:26.546126 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:26.546087    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 13:33:24 +0000 UTC" deadline="2028-04-18 01:06:39.396409821 +0000 UTC"
Sep 15 13:38:26.546126 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:26.546125    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13931h28m12.850289812s"
Sep 15 13:38:26.639755 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:26.639720    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:26.639981 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:26.639890    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:26.639981 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:26.639949    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:26.640126 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:26.640085    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:26.640194 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:26.640163    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:26.640287 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:26.640269    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:28.142977 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:28.142945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdmc\" (UniqueName: \"kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc\") pod \"network-check-target-j5pmd\" (UID: \"ef84ca8e-d344-4d38-b015-8a76a03e5dfa\") " pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:28.143594 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:28.143097    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 13:38:28.143594 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:28.143112    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 13:38:28.143594 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:28.143120    2582 projected.go:196] Error preparing data for projected volume kube-api-access-sbdmc for pod openshift-network-diagnostics/network-check-target-j5pmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:28.143594 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:28.143191    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc podName:ef84ca8e-d344-4d38-b015-8a76a03e5dfa nodeName:}" failed. No retries permitted until 2026-09-15 13:38:32.143168683 +0000 UTC m=+10.021071503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbdmc" (UniqueName: "kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc") pod "network-check-target-j5pmd" (UID: "ef84ca8e-d344-4d38-b015-8a76a03e5dfa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:28.244476 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:28.244120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs\") pod \"network-metrics-daemon-5kxxr\" (UID: \"0258dc54-5388-49fa-bd24-0ee885930283\") " pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:28.244476 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:28.244241    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:28.244476 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:28.244257    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:28.244476 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:28.244316    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs podName:0258dc54-5388-49fa-bd24-0ee885930283 nodeName:}" failed. No retries permitted until 2026-09-15 13:38:32.244299539 +0000 UTC m=+10.122202353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs") pod "network-metrics-daemon-5kxxr" (UID: "0258dc54-5388-49fa-bd24-0ee885930283") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:28.244476 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:28.244395    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:28.244476 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:28.244458    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret podName:b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe nodeName:}" failed. No retries permitted until 2026-09-15 13:38:32.244443001 +0000 UTC m=+10.122345813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret") pod "global-pull-secret-syncer-fbj84" (UID: "b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe") : object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:28.637740 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:28.637221    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:28.637740 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:28.637366    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:28.637740 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:28.637428    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:28.637740 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:28.637508    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:28.637740 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:28.637625    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:28.637740 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:28.637705    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:30.640681 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:30.639845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:30.640681 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:30.639987    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:30.640681 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:30.640382    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:30.640681 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:30.640485    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:30.640681 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:30.640539    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:30.640681 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:30.640610    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:32.183387 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:32.183355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdmc\" (UniqueName: \"kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc\") pod \"network-check-target-j5pmd\" (UID: \"ef84ca8e-d344-4d38-b015-8a76a03e5dfa\") " pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:32.183810 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:32.183578    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 13:38:32.183810 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:32.183600    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 13:38:32.183810 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:32.183613    2582 projected.go:196] Error preparing data for projected volume kube-api-access-sbdmc for pod openshift-network-diagnostics/network-check-target-j5pmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:32.183810 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:32.183677    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc podName:ef84ca8e-d344-4d38-b015-8a76a03e5dfa nodeName:}" failed. No retries permitted until 2026-09-15 13:38:40.1836596 +0000 UTC m=+18.061562622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbdmc" (UniqueName: "kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc") pod "network-check-target-j5pmd" (UID: "ef84ca8e-d344-4d38-b015-8a76a03e5dfa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:32.284091 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:32.284053    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs\") pod \"network-metrics-daemon-5kxxr\" (UID: \"0258dc54-5388-49fa-bd24-0ee885930283\") " pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:32.284278 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:32.284111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:32.284278 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:32.284249    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:32.284353 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:32.284314    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret podName:b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe nodeName:}" failed. No retries permitted until 2026-09-15 13:38:40.284301094 +0000 UTC m=+18.162203891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret") pod "global-pull-secret-syncer-fbj84" (UID: "b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe") : object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:32.284675 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:32.284653    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:32.284759 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:32.284717    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs podName:0258dc54-5388-49fa-bd24-0ee885930283 nodeName:}" failed. No retries permitted until 2026-09-15 13:38:40.284704276 +0000 UTC m=+18.162607073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs") pod "network-metrics-daemon-5kxxr" (UID: "0258dc54-5388-49fa-bd24-0ee885930283") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:32.638944 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:32.637991    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:32.638944 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:32.638105    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:32.638944 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:32.638564    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:32.638944 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:32.638671    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:32.638944 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:32.638740    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:32.638944 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:32.638811    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:34.637072 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:34.636912    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:34.637072 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:34.636958    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:34.637072 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:34.636912    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:34.637072 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:34.637032    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:34.637863 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:34.637134    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:34.637863 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:34.637287    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:36.636871 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:36.636841    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:36.637362 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:36.636839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:36.637362 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:36.636961    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:36.637362 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:36.636847    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:36.637362 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:36.637050    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:36.637362 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:36.637149    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:38.637284 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:38.637247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:38.637873 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:38.637289    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:38.637873 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:38.637248    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:38.637873 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:38.637382    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:38.637873 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:38.637462    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:38.637873 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:38.637535    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:40.242029 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:40.241988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdmc\" (UniqueName: \"kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc\") pod \"network-check-target-j5pmd\" (UID: \"ef84ca8e-d344-4d38-b015-8a76a03e5dfa\") " pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:40.242556 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:40.242159    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 13:38:40.242556 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:40.242187    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 13:38:40.242556 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:40.242201    2582 projected.go:196] Error preparing data for projected volume kube-api-access-sbdmc for pod openshift-network-diagnostics/network-check-target-j5pmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:40.242556 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:40.242264    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc podName:ef84ca8e-d344-4d38-b015-8a76a03e5dfa nodeName:}" failed. No retries permitted until 2026-09-15 13:38:56.242246319 +0000 UTC m=+34.120149124 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbdmc" (UniqueName: "kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc") pod "network-check-target-j5pmd" (UID: "ef84ca8e-d344-4d38-b015-8a76a03e5dfa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:40.343421 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:40.343389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs\") pod \"network-metrics-daemon-5kxxr\" (UID: \"0258dc54-5388-49fa-bd24-0ee885930283\") " pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:40.343421 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:40.343427    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:40.343634 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:40.343554    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:40.343634 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:40.343566    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:40.343634 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:40.343618    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs podName:0258dc54-5388-49fa-bd24-0ee885930283 nodeName:}" failed. No retries permitted until 2026-09-15 13:38:56.343601521 +0000 UTC m=+34.221504318 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs") pod "network-metrics-daemon-5kxxr" (UID: "0258dc54-5388-49fa-bd24-0ee885930283") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:40.343634 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:40.343633    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret podName:b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe nodeName:}" failed. No retries permitted until 2026-09-15 13:38:56.343626472 +0000 UTC m=+34.221529269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret") pod "global-pull-secret-syncer-fbj84" (UID: "b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe") : object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:40.636798 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:40.636759    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:40.636798 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:40.636785    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:40.637045 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:40.636877    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:40.637045 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:40.636966    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:40.637167 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:40.637076    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:40.637220 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:40.637190    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:42.637453 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:42.637417    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:42.637893 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:42.637513    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:42.637893 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:42.637642    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:42.637893 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:42.637521    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:42.637893 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:42.637703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:42.637893 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:42.637824    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:43.704757 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.704417    2582 generic.go:356] "Generic (PLEG): container finished" podID="76b83e804e1c7d5c53a2939b9adbecc1" containerID="ce86a6a256eb303e8228654b8a2088c480138ebf866595f3ec628908a78c4919" exitCode=0
Sep 15 13:38:43.704757 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.704505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" event={"ID":"76b83e804e1c7d5c53a2939b9adbecc1","Type":"ContainerDied","Data":"ce86a6a256eb303e8228654b8a2088c480138ebf866595f3ec628908a78c4919"}
Sep 15 13:38:43.706472 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.706445    2582 generic.go:356] "Generic (PLEG): container finished" podID="e42c6f90-64b3-4a55-bf82-c9ee9a126dc9" containerID="72531fd42f28d02850d62d63250fd34cf8dca4a6fc6f1b1370ff09585a7a252a" exitCode=0
Sep 15 13:38:43.706591 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.706527    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxgrp" event={"ID":"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9","Type":"ContainerDied","Data":"72531fd42f28d02850d62d63250fd34cf8dca4a6fc6f1b1370ff09585a7a252a"}
Sep 15 13:38:43.708214 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.708185    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b7tq6" event={"ID":"7edf22ea-7f60-4d71-95db-8594c103b405","Type":"ContainerStarted","Data":"4fe346cc6a7c8875f4f99253943b89aaacda9175ef95db4617b8538e35012943"}
Sep 15 13:38:43.709817 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.709795    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lhd7g" event={"ID":"9a1476f2-df89-474d-bbce-2258e2487080","Type":"ContainerStarted","Data":"fdac22bf45fea01065e4dfbd558fdb062d91fe3f95ef26e77a5e56ce9b18e423"}
Sep 15 13:38:43.713160 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.713119    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl" event={"ID":"464b7bb4-eaaa-4752-9055-3284a71d4e48","Type":"ContainerStarted","Data":"0213d3529be03ab7ac96d38a6f1a12137777ddb1933e52545d7e3ee347a8769a"}
Sep 15 13:38:43.713254 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.713198    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl" event={"ID":"464b7bb4-eaaa-4752-9055-3284a71d4e48","Type":"ContainerStarted","Data":"876e389cb79d00e0bbee84cd69501e5b706c7f3d43e3c25b210ed6a13320a672"}
Sep 15 13:38:43.713254 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.713249    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl" event={"ID":"464b7bb4-eaaa-4752-9055-3284a71d4e48","Type":"ContainerStarted","Data":"a9803e562665340600828e2db09fe30760eaeca2715aae862be8f4fe3033f77e"}
Sep 15 13:38:43.713327 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.713280    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl" event={"ID":"464b7bb4-eaaa-4752-9055-3284a71d4e48","Type":"ContainerStarted","Data":"0f189bff0098f46812dc42fba279c3b69779a88a6f2a68216923cab97711b38e"}
Sep 15 13:38:43.714598 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.714577    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-92nmn" event={"ID":"ac943d06-bf47-4dbf-8d40-8c63e0c48007","Type":"ContainerStarted","Data":"d560256b17d9dcf815b61440b7387868259cde8627b6ba38186d2e8baa60cb82"}
Sep 15 13:38:43.716082 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.716062    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s2l72" event={"ID":"ae90d50d-e61a-43d9-8286-dc01dc000417","Type":"ContainerStarted","Data":"d092ff27cae763aabc75a7b19be6690cda816258c670ed0b58b4ee4b6025e7ca"}
Sep 15 13:38:43.717538 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.717521    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7c5tp" event={"ID":"5ed7f5fd-fd66-4f31-a0e8-f6453fd55829","Type":"ContainerStarted","Data":"2f8749cf91f42fd224450a79c4f64b83a5e4b7f139924886e7a9273ee91ce459"}
Sep 15 13:38:43.718970 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.718952    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt" event={"ID":"9b3b9786-6a4f-4c44-8eec-6602a1acd59d","Type":"ContainerStarted","Data":"df64e5703b49d703c44b9884c8e933ab893798ba0ec9b27fa49c2df98c36eb60"}
Sep 15 13:38:43.732449 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.732399    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b7tq6" podStartSLOduration=2.926677724 podStartE2EDuration="20.732377976s" totalImagesPullingTime="17.805700252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:38:43.732013744 +0000 UTC m=+21.609916578" watchObservedRunningTime="2026-09-15 13:38:43.732377976 +0000 UTC m=+21.610280798"
Sep 15 13:38:43.745068 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.745017    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-92nmn" podStartSLOduration=3.910209097 podStartE2EDuration="21.744997722s" totalImagesPullingTime="17.834788625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:38:43.744500503 +0000 UTC m=+21.622403323" watchObservedRunningTime="2026-09-15 13:38:43.744997722 +0000 UTC m=+21.622900546"
Sep 15 13:38:43.777334 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.777291    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lhd7g" podStartSLOduration=3.962872071 podStartE2EDuration="21.777275836s" totalImagesPullingTime="17.814403765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:38:43.777219497 +0000 UTC m=+21.655122316" watchObservedRunningTime="2026-09-15 13:38:43.777275836 +0000 UTC m=+21.655178655"
Sep 15 13:38:43.777465 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.777414    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s2l72" podStartSLOduration=3.883164168 podStartE2EDuration="21.777403882s" totalImagesPullingTime="17.894239714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:38:43.762621227 +0000 UTC m=+21.640524047" watchObservedRunningTime="2026-09-15 13:38:43.777403882 +0000 UTC m=+21.655306704"
Sep 15 13:38:43.796342 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:43.796292    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7c5tp" podStartSLOduration=3.950392325 podStartE2EDuration="21.796272891s" totalImagesPullingTime="17.845880566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:38:43.795979241 +0000 UTC m=+21.673882061" watchObservedRunningTime="2026-09-15 13:38:43.796272891 +0000 UTC m=+21.674175710"
Sep 15 13:38:44.636697 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:44.636664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:44.636953 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:44.636774    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:44.636953 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:44.636775    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:44.636953 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:44.636884    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:44.637071 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:44.636961    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:44.637071 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:44.636869    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:44.723590 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:44.723555    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl" event={"ID":"464b7bb4-eaaa-4752-9055-3284a71d4e48","Type":"ContainerStarted","Data":"ecece0b9a8db3f5704df165ca9bc472f59397cbf4593b5e79b75285b19f8b2b9"}
Sep 15 13:38:44.724197 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:44.723628    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl" event={"ID":"464b7bb4-eaaa-4752-9055-3284a71d4e48","Type":"ContainerStarted","Data":"7cacd64311fe086ef52e6dd215b1c6074bfbb2042096b3614f9233ab27bd4b97"}
Sep 15 13:38:44.725002 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:44.724980    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zlxcv" event={"ID":"fae7c57f-7dfa-47ae-b61a-6fc3d8eaf103","Type":"ContainerStarted","Data":"ab2cc88b99f1fd9b315a6702bbaa75af9ec8291647530a4c630a6a372af2f623"}
Sep 15 13:38:44.726721 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:44.726694    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" event={"ID":"76b83e804e1c7d5c53a2939b9adbecc1","Type":"ContainerStarted","Data":"87213a203147b1c94bf16c02ec95d1b24afb725cc0d4915f5b729a08d4012559"}
Sep 15 13:38:44.740524 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:44.740474    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zlxcv" podStartSLOduration=4.955739036 podStartE2EDuration="22.740460527s" totalImagesPullingTime="17.784721491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:38:44.73989672 +0000 UTC m=+22.617799542" watchObservedRunningTime="2026-09-15 13:38:44.740460527 +0000 UTC m=+22.618363346"
Sep 15 13:38:44.754879 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:44.754838    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" podStartSLOduration=21.754823698 podStartE2EDuration="21.754823698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:38:44.753948882 +0000 UTC m=+22.631851723" watchObservedRunningTime="2026-09-15 13:38:44.754823698 +0000 UTC m=+22.632726517"
Sep 15 13:38:45.885037 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:45.884910    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 13:38:46.531607 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:46.531331    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-92nmn"
Sep 15 13:38:46.532384 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:46.532361    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-92nmn"
Sep 15 13:38:46.563213 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:46.562980    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T13:38:45.884926507Z","UUID":"737fd854-f9af-4b2c-999c-15f7f4ba19d1","Handler":null,"Name":"","Endpoint":""}
Sep 15 13:38:46.564616 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:46.564591    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 13:38:46.564748 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:46.564632    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 13:38:46.636506 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:46.636479    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:46.636506 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:46.636500    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:46.636739 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:46.636598    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:46.636739 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:46.636678    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:46.636739 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:46.636710    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:46.636888 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:46.636785    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:46.736724 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:46.736672    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl" event={"ID":"464b7bb4-eaaa-4752-9055-3284a71d4e48","Type":"ContainerStarted","Data":"d17bbcc19c8e012d7ee1db8325864b5649628d87466769b66c09d1ffe217f683"}
Sep 15 13:38:46.739238 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:46.739159    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt" event={"ID":"9b3b9786-6a4f-4c44-8eec-6602a1acd59d","Type":"ContainerStarted","Data":"7ef7ba5fc0ef4619e2ed5b65265b1faa082087531951e11c83e4a6ec3fe1ae63"}
Sep 15 13:38:46.739433 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:46.739407    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-92nmn"
Sep 15 13:38:46.739930 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:46.739903    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-92nmn"
Sep 15 13:38:47.744007 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:47.743968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt" event={"ID":"9b3b9786-6a4f-4c44-8eec-6602a1acd59d","Type":"ContainerStarted","Data":"02b1b17e4fcfa50161c4acacf0749460fb2643b5693cbd1ac8062a81016770f1"}
Sep 15 13:38:47.762421 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:47.762374    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqtxt" podStartSLOduration=4.791911766 podStartE2EDuration="25.76235879s" totalImagesPullingTime="20.970447024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:38:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:38:47.762332521 +0000 UTC m=+25.640235341" watchObservedRunningTime="2026-09-15 13:38:47.76235879 +0000 UTC m=+25.640261586"
Sep 15 13:38:48.637396 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:48.637364    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:48.637565 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:48.637374    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:48.637565 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:48.637488    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:48.637565 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:48.637495    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:48.637682 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:48.637602    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:48.637717 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:48.637683    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:49.750618 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:49.750584    2582 generic.go:356] "Generic (PLEG): container finished" podID="e42c6f90-64b3-4a55-bf82-c9ee9a126dc9" containerID="7f613eed1957c1fd0007440eb6750a514dee6162be2f8eea0aa6d0f7c407e23b" exitCode=0
Sep 15 13:38:49.751035 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:49.750662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxgrp" event={"ID":"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9","Type":"ContainerDied","Data":"7f613eed1957c1fd0007440eb6750a514dee6162be2f8eea0aa6d0f7c407e23b"}
Sep 15 13:38:49.754774 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:49.754747    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl" event={"ID":"464b7bb4-eaaa-4752-9055-3284a71d4e48","Type":"ContainerStarted","Data":"f293f58f1686b411c988fc2a1976cb5565f022303c5ff8e63c14b0bab8b3b4a8"}
Sep 15 13:38:49.754977 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:49.754963    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:49.755051 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:49.755011    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:49.770460 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:49.770433    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:49.802757 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:49.802708    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl" podStartSLOduration=9.938210524 podStartE2EDuration="27.802693204s" totalImagesPullingTime="17.86448268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:38:49.802387425 +0000 UTC m=+27.680290244" watchObservedRunningTime="2026-09-15 13:38:49.802693204 +0000 UTC m=+27.680596022"
Sep 15 13:38:50.637341 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:50.637312    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:50.637341 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:50.637328    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:50.637680 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:50.637321    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:50.637680 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:50.637451    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:50.637680 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:50.637472    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:50.637680 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:50.637560    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:50.757511 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:50.757482    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:50.772560 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:50.772534    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:38:50.996539 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:50.996463    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5pmd"]
Sep 15 13:38:50.996684 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:50.996585    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:50.996738 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:50.996710    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:51.000067 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:51.000035    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fbj84"]
Sep 15 13:38:51.000224 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:51.000186    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:51.000301 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:51.000281    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:51.000871 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:51.000850    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5kxxr"]
Sep 15 13:38:51.000960 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:51.000946    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:51.001072 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:51.001048    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:52.637464 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:52.637438    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:52.637971 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:52.637521    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:52.637971 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:52.637817    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:52.637971 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:52.637873    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:52.637971 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:52.637907    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:52.637971 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:52.637945    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:52.764021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:52.763844    2582 generic.go:356] "Generic (PLEG): container finished" podID="e42c6f90-64b3-4a55-bf82-c9ee9a126dc9" containerID="9bd72c8e7f74414a94212d6e0be10ab6cfa121250f15fe1a8df34a52aa984752" exitCode=0
Sep 15 13:38:52.764204 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:52.763927    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxgrp" event={"ID":"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9","Type":"ContainerDied","Data":"9bd72c8e7f74414a94212d6e0be10ab6cfa121250f15fe1a8df34a52aa984752"}
Sep 15 13:38:53.769477 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:53.769452    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxgrp" event={"ID":"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9","Type":"ContainerStarted","Data":"be892934742dadd266add28ddb4e104a4f30108ed3647fc9fd96e2a03ee8218f"}
Sep 15 13:38:54.636804 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:54.636771    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:54.636804 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:54.636796    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:54.637042 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:54.636771    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:54.637042 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:54.636893    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:54.637042 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:54.636967    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:54.637198 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:54.637042    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:54.773454 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:54.773421    2582 generic.go:356] "Generic (PLEG): container finished" podID="e42c6f90-64b3-4a55-bf82-c9ee9a126dc9" containerID="be892934742dadd266add28ddb4e104a4f30108ed3647fc9fd96e2a03ee8218f" exitCode=0
Sep 15 13:38:54.773801 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:54.773478    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxgrp" event={"ID":"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9","Type":"ContainerDied","Data":"be892934742dadd266add28ddb4e104a4f30108ed3647fc9fd96e2a03ee8218f"}
Sep 15 13:38:56.254834 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:56.254799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdmc\" (UniqueName: \"kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc\") pod \"network-check-target-j5pmd\" (UID: \"ef84ca8e-d344-4d38-b015-8a76a03e5dfa\") " pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:56.255249 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:56.254954    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 13:38:56.255249 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:56.254971    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 13:38:56.255249 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:56.254979    2582 projected.go:196] Error preparing data for projected volume kube-api-access-sbdmc for pod openshift-network-diagnostics/network-check-target-j5pmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:56.255249 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:56.255036    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc podName:ef84ca8e-d344-4d38-b015-8a76a03e5dfa nodeName:}" failed. No retries permitted until 2026-09-15 13:39:28.255015789 +0000 UTC m=+66.132918589 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbdmc" (UniqueName: "kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc") pod "network-check-target-j5pmd" (UID: "ef84ca8e-d344-4d38-b015-8a76a03e5dfa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 13:38:56.355798 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:56.355757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs\") pod \"network-metrics-daemon-5kxxr\" (UID: \"0258dc54-5388-49fa-bd24-0ee885930283\") " pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:56.355798 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:56.355801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:56.356033 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:56.355903    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:56.356033 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:56.355914    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:56.356033 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:56.355969    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret podName:b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe nodeName:}" failed. No retries permitted until 2026-09-15 13:39:28.355953132 +0000 UTC m=+66.233855952 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret") pod "global-pull-secret-syncer-fbj84" (UID: "b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe") : object "kube-system"/"original-pull-secret" not registered
Sep 15 13:38:56.356033 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:56.355987    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs podName:0258dc54-5388-49fa-bd24-0ee885930283 nodeName:}" failed. No retries permitted until 2026-09-15 13:39:28.355977898 +0000 UTC m=+66.233880698 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs") pod "network-metrics-daemon-5kxxr" (UID: "0258dc54-5388-49fa-bd24-0ee885930283") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 13:38:56.637437 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:56.637343    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:56.637635 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:56.637347    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:56.637635 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:56.637492    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5pmd" podUID="ef84ca8e-d344-4d38-b015-8a76a03e5dfa"
Sep 15 13:38:56.637635 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:56.637347    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:56.637635 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:56.637596    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fbj84" podUID="b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe"
Sep 15 13:38:56.637774 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:56.637679    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kxxr" podUID="0258dc54-5388-49fa-bd24-0ee885930283"
Sep 15 13:38:56.892811 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:56.892731    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeReady"
Sep 15 13:38:56.892975 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:56.892899    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 13:38:56.954122 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:56.954081    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57f75d7c55-whpk5"]
Sep 15 13:38:56.999938 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:56.999895    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"]
Sep 15 13:38:57.000109 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:56.999987    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.002973 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.002780    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kxsf6\""
Sep 15 13:38:57.002973 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.002782    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 13:38:57.002973 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.002844    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 13:38:57.002973 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.002856    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 13:38:57.013363 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.013249    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"]
Sep 15 13:38:57.013476 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.013379    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zp7l2"]
Sep 15 13:38:57.015976 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.015948    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 13:38:57.016262 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.016243    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"
Sep 15 13:38:57.018567 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.018544    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 15 13:38:57.019062 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.019042    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 15 13:38:57.019217 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.019196    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 15 13:38:57.019448 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.019423    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pbxlj\""
Sep 15 13:38:57.019448 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.019432    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 15 13:38:57.035087 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.035021    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"]
Sep 15 13:38:57.035225 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.035134    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:57.037374 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.037302    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 13:38:57.037374 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.037352    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 13:38:57.037374 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.037370    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dfkpb\""
Sep 15 13:38:57.053178 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.053155    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-xwgbl"]
Sep 15 13:38:57.053292 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.053274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:38:57.056672 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.056304    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 15 13:38:57.056672 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.056336    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-r9fql\""
Sep 15 13:38:57.056672 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.056546    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 15 13:38:57.060929 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.060910    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-bound-sa-token\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.061069 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.060945    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa7df37b-0f61-4c8e-849c-c680261dc753-trusted-ca\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.061069 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.061008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa7df37b-0f61-4c8e-849c-c680261dc753-image-registry-private-configuration\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.061069 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.061042    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.061227 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.061071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-certificates\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.061227 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.061097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa7df37b-0f61-4c8e-849c-c680261dc753-installation-pull-secrets\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.061227 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.061171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa7df37b-0f61-4c8e-849c-c680261dc753-ca-trust-extracted\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.061227 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.061212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz5pq\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-kube-api-access-tz5pq\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.066862 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.066844    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cqqkh"]
Sep 15 13:38:57.067025 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.067007    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-xwgbl"
Sep 15 13:38:57.069178 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.069158    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 13:38:57.069367 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.069343    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 13:38:57.069504 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.069484    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-4t4d9\""
Sep 15 13:38:57.071422 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.071402    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57f75d7c55-whpk5"]
Sep 15 13:38:57.071515 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.071432    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-xwgbl"]
Sep 15 13:38:57.071515 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.071446    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"]
Sep 15 13:38:57.071515 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.071464    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cqqkh"]
Sep 15 13:38:57.071515 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.071483    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zp7l2"]
Sep 15 13:38:57.071812 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.071793    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cqqkh"
Sep 15 13:38:57.074033 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.074014    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mmgq5\""
Sep 15 13:38:57.075171 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.075052    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 13:38:57.075171 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.075066    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 13:38:57.075171 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.075083    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 13:38:57.161697 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa7df37b-0f61-4c8e-849c-c680261dc753-image-registry-private-configuration\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.161697 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34ab498e-e299-4cab-85a5-d295962eadc8-config\") pod \"service-ca-operator-cf7c776b7-rrr44\" (UID: \"34ab498e-e299-4cab-85a5-d295962eadc8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"
Sep 15 13:38:57.161697 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161675    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8af82a-61dd-44cb-a4a6-27bba6accd04-config-volume\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:57.161697 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161690    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8af82a-61dd-44cb-a4a6-27bba6accd04-tmp-dir\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:57.161997 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161712    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4884128c-7457-4add-85ce-103712920e9b-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-kt4ds\" (UID: \"4884128c-7457-4add-85ce-103712920e9b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:38:57.161997 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaa9a50a-44c0-4d90-9a3b-1855ab51b369-cert\") pod \"ingress-canary-cqqkh\" (UID: \"aaa9a50a-44c0-4d90-9a3b-1855ab51b369\") " pod="openshift-ingress-canary/ingress-canary-cqqkh"
Sep 15 13:38:57.161997 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzwl2\" (UniqueName: \"kubernetes.io/projected/d6d042ce-39af-404c-865d-440addce75f9-kube-api-access-bzwl2\") pod \"network-check-source-6f8b768455-xwgbl\" (UID: \"d6d042ce-39af-404c-865d-440addce75f9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-xwgbl"
Sep 15 13:38:57.161997 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.161997 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-certificates\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.161997 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrtqd\" (UniqueName: \"kubernetes.io/projected/ea8af82a-61dd-44cb-a4a6-27bba6accd04-kube-api-access-vrtqd\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:57.161997 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161882    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa7df37b-0f61-4c8e-849c-c680261dc753-installation-pull-secrets\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.161997 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161931    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kt4ds\" (UID: \"4884128c-7457-4add-85ce-103712920e9b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:38:57.161997 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa7df37b-0f61-4c8e-849c-c680261dc753-ca-trust-extracted\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.161997 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.161998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz5pq\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-kube-api-access-tz5pq\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.162473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.162027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2wms\" (UniqueName: \"kubernetes.io/projected/34ab498e-e299-4cab-85a5-d295962eadc8-kube-api-access-h2wms\") pod \"service-ca-operator-cf7c776b7-rrr44\" (UID: \"34ab498e-e299-4cab-85a5-d295962eadc8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"
Sep 15 13:38:57.162473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.162060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8njf6\" (UniqueName: \"kubernetes.io/projected/aaa9a50a-44c0-4d90-9a3b-1855ab51b369-kube-api-access-8njf6\") pod \"ingress-canary-cqqkh\" (UID: \"aaa9a50a-44c0-4d90-9a3b-1855ab51b369\") " pod="openshift-ingress-canary/ingress-canary-cqqkh"
Sep 15 13:38:57.162473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.162103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:57.162473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.162179    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-bound-sa-token\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.162473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.162227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa7df37b-0f61-4c8e-849c-c680261dc753-trusted-ca\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.162473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.162250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/34ab498e-e299-4cab-85a5-d295962eadc8-serving-cert\") pod \"service-ca-operator-cf7c776b7-rrr44\" (UID: \"34ab498e-e299-4cab-85a5-d295962eadc8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"
Sep 15 13:38:57.162473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.162435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa7df37b-0f61-4c8e-849c-c680261dc753-ca-trust-extracted\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.162473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.162441    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-certificates\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.162473 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.162437    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 13:38:57.162473 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.162464    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57f75d7c55-whpk5: secret "image-registry-tls" not found
Sep 15 13:38:57.162880 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.162595    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls podName:aa7df37b-0f61-4c8e-849c-c680261dc753 nodeName:}" failed. No retries permitted until 2026-09-15 13:38:57.662574936 +0000 UTC m=+35.540477736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls") pod "image-registry-57f75d7c55-whpk5" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753") : secret "image-registry-tls" not found
Sep 15 13:38:57.163329 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.163307    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa7df37b-0f61-4c8e-849c-c680261dc753-trusted-ca\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.166022 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.165994    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:38:57.166499 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.166482    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa7df37b-0f61-4c8e-849c-c680261dc753-installation-pull-secrets\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.166586 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.166541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa7df37b-0f61-4c8e-849c-c680261dc753-image-registry-private-configuration\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.171128 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.171106    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-bound-sa-token\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.171305 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.171285    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz5pq\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-kube-api-access-tz5pq\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.246135 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.246101    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:38:57.263181 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:57.263534 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/34ab498e-e299-4cab-85a5-d295962eadc8-serving-cert\") pod \"service-ca-operator-cf7c776b7-rrr44\" (UID: \"34ab498e-e299-4cab-85a5-d295962eadc8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"
Sep 15 13:38:57.263534 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34ab498e-e299-4cab-85a5-d295962eadc8-config\") pod \"service-ca-operator-cf7c776b7-rrr44\" (UID: \"34ab498e-e299-4cab-85a5-d295962eadc8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"
Sep 15 13:38:57.263534 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8af82a-61dd-44cb-a4a6-27bba6accd04-config-volume\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:57.263534 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263280    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8af82a-61dd-44cb-a4a6-27bba6accd04-tmp-dir\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:57.263534 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4884128c-7457-4add-85ce-103712920e9b-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-kt4ds\" (UID: \"4884128c-7457-4add-85ce-103712920e9b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:38:57.263534 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.263298    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 13:38:57.263534 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaa9a50a-44c0-4d90-9a3b-1855ab51b369-cert\") pod \"ingress-canary-cqqkh\" (UID: \"aaa9a50a-44c0-4d90-9a3b-1855ab51b369\") " pod="openshift-ingress-canary/ingress-canary-cqqkh"
Sep 15 13:38:57.263534 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzwl2\" (UniqueName: \"kubernetes.io/projected/d6d042ce-39af-404c-865d-440addce75f9-kube-api-access-bzwl2\") pod \"network-check-source-6f8b768455-xwgbl\" (UID: \"d6d042ce-39af-404c-865d-440addce75f9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-xwgbl"
Sep 15 13:38:57.263534 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.263379    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls podName:ea8af82a-61dd-44cb-a4a6-27bba6accd04 nodeName:}" failed. No retries permitted until 2026-09-15 13:38:57.763357889 +0000 UTC m=+35.641260711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls") pod "dns-default-zp7l2" (UID: "ea8af82a-61dd-44cb-a4a6-27bba6accd04") : secret "dns-default-metrics-tls" not found
Sep 15 13:38:57.263982 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263729    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8af82a-61dd-44cb-a4a6-27bba6accd04-tmp-dir\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:57.263982 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrtqd\" (UniqueName: \"kubernetes.io/projected/ea8af82a-61dd-44cb-a4a6-27bba6accd04-kube-api-access-vrtqd\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:57.263982 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263831    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kt4ds\" (UID: \"4884128c-7457-4add-85ce-103712920e9b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:38:57.263982 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2wms\" (UniqueName: \"kubernetes.io/projected/34ab498e-e299-4cab-85a5-d295962eadc8-kube-api-access-h2wms\") pod \"service-ca-operator-cf7c776b7-rrr44\" (UID: \"34ab498e-e299-4cab-85a5-d295962eadc8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"
Sep 15 13:38:57.263982 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8njf6\" (UniqueName: \"kubernetes.io/projected/aaa9a50a-44c0-4d90-9a3b-1855ab51b369-kube-api-access-8njf6\") pod \"ingress-canary-cqqkh\" (UID: \"aaa9a50a-44c0-4d90-9a3b-1855ab51b369\") " pod="openshift-ingress-canary/ingress-canary-cqqkh"
Sep 15 13:38:57.264240 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.263998    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8af82a-61dd-44cb-a4a6-27bba6accd04-config-volume\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:57.264240 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.264036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34ab498e-e299-4cab-85a5-d295962eadc8-config\") pod \"service-ca-operator-cf7c776b7-rrr44\" (UID: \"34ab498e-e299-4cab-85a5-d295962eadc8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"
Sep 15 13:38:57.264240 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.264072    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 13:38:57.264240 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.264120    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert podName:4884128c-7457-4add-85ce-103712920e9b nodeName:}" failed. No retries permitted until 2026-09-15 13:38:57.764105556 +0000 UTC m=+35.642008362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-kt4ds" (UID: "4884128c-7457-4add-85ce-103712920e9b") : secret "networking-console-plugin-cert" not found
Sep 15 13:38:57.264418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.264293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4884128c-7457-4add-85ce-103712920e9b-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-kt4ds\" (UID: \"4884128c-7457-4add-85ce-103712920e9b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:38:57.266241 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.266218    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/34ab498e-e299-4cab-85a5-d295962eadc8-serving-cert\") pod \"service-ca-operator-cf7c776b7-rrr44\" (UID: \"34ab498e-e299-4cab-85a5-d295962eadc8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"
Sep 15 13:38:57.266699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.266682    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaa9a50a-44c0-4d90-9a3b-1855ab51b369-cert\") pod \"ingress-canary-cqqkh\" (UID: \"aaa9a50a-44c0-4d90-9a3b-1855ab51b369\") " pod="openshift-ingress-canary/ingress-canary-cqqkh"
Sep 15 13:38:57.273730 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.273694    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrtqd\" (UniqueName: \"kubernetes.io/projected/ea8af82a-61dd-44cb-a4a6-27bba6accd04-kube-api-access-vrtqd\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:57.274543 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.274521    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2wms\" (UniqueName: \"kubernetes.io/projected/34ab498e-e299-4cab-85a5-d295962eadc8-kube-api-access-h2wms\") pod \"service-ca-operator-cf7c776b7-rrr44\" (UID: \"34ab498e-e299-4cab-85a5-d295962eadc8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"
Sep 15 13:38:57.275279 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.275256    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzwl2\" (UniqueName: \"kubernetes.io/projected/d6d042ce-39af-404c-865d-440addce75f9-kube-api-access-bzwl2\") pod \"network-check-source-6f8b768455-xwgbl\" (UID: \"d6d042ce-39af-404c-865d-440addce75f9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-xwgbl"
Sep 15 13:38:57.276513 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.276490    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8njf6\" (UniqueName: \"kubernetes.io/projected/aaa9a50a-44c0-4d90-9a3b-1855ab51b369-kube-api-access-8njf6\") pod \"ingress-canary-cqqkh\" (UID: \"aaa9a50a-44c0-4d90-9a3b-1855ab51b369\") " pod="openshift-ingress-canary/ingress-canary-cqqkh"
Sep 15 13:38:57.327527 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.327494    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"
Sep 15 13:38:57.379292 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.379258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-xwgbl"
Sep 15 13:38:57.385440 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.385409    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cqqkh"
Sep 15 13:38:57.656067 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.655746    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44"]
Sep 15 13:38:57.659671 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:57.659633    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ab498e_e299_4cab_85a5_d295962eadc8.slice/crio-a0db117400ae618b7c01b54a148096482f4c1651ebcaff34ce84c0d14a149306 WatchSource:0}: Error finding container a0db117400ae618b7c01b54a148096482f4c1651ebcaff34ce84c0d14a149306: Status 404 returned error can't find the container with id a0db117400ae618b7c01b54a148096482f4c1651ebcaff34ce84c0d14a149306
Sep 15 13:38:57.667941 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.667911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:57.668082 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.668069    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 13:38:57.668121 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.668084    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57f75d7c55-whpk5: secret "image-registry-tls" not found
Sep 15 13:38:57.668175 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.668153    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls podName:aa7df37b-0f61-4c8e-849c-c680261dc753 nodeName:}" failed. No retries permitted until 2026-09-15 13:38:58.66811431 +0000 UTC m=+36.546017127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls") pod "image-registry-57f75d7c55-whpk5" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753") : secret "image-registry-tls" not found
Sep 15 13:38:57.692530 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.692500    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-xwgbl"]
Sep 15 13:38:57.696094 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:57.696058    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d042ce_39af_404c_865d_440addce75f9.slice/crio-580c8a86efbdaa0500e35db4255edb1f3114ed67135377004ef688d51f08c671 WatchSource:0}: Error finding container 580c8a86efbdaa0500e35db4255edb1f3114ed67135377004ef688d51f08c671: Status 404 returned error can't find the container with id 580c8a86efbdaa0500e35db4255edb1f3114ed67135377004ef688d51f08c671
Sep 15 13:38:57.713653 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.713625    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cqqkh"]
Sep 15 13:38:57.717246 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:38:57.717133    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa9a50a_44c0_4d90_9a3b_1855ab51b369.slice/crio-58c975ec962419897dbf7437db78777f69178bec2240144a71e08d3e3e4e0995 WatchSource:0}: Error finding container 58c975ec962419897dbf7437db78777f69178bec2240144a71e08d3e3e4e0995: Status 404 returned error can't find the container with id 58c975ec962419897dbf7437db78777f69178bec2240144a71e08d3e3e4e0995
Sep 15 13:38:57.768982 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.768953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kt4ds\" (UID: \"4884128c-7457-4add-85ce-103712920e9b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:38:57.769161 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.768999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:57.769161 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.769089    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 13:38:57.769161 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.769097    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 13:38:57.769161 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.769152    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls podName:ea8af82a-61dd-44cb-a4a6-27bba6accd04 nodeName:}" failed. No retries permitted until 2026-09-15 13:38:58.769126256 +0000 UTC m=+36.647029060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls") pod "dns-default-zp7l2" (UID: "ea8af82a-61dd-44cb-a4a6-27bba6accd04") : secret "dns-default-metrics-tls" not found
Sep 15 13:38:57.769316 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:57.769170    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert podName:4884128c-7457-4add-85ce-103712920e9b nodeName:}" failed. No retries permitted until 2026-09-15 13:38:58.769162102 +0000 UTC m=+36.647064899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-kt4ds" (UID: "4884128c-7457-4add-85ce-103712920e9b") : secret "networking-console-plugin-cert" not found
Sep 15 13:38:57.781597 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.781565    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-xwgbl" event={"ID":"d6d042ce-39af-404c-865d-440addce75f9","Type":"ContainerStarted","Data":"580c8a86efbdaa0500e35db4255edb1f3114ed67135377004ef688d51f08c671"}
Sep 15 13:38:57.782619 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.782600    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cqqkh" event={"ID":"aaa9a50a-44c0-4d90-9a3b-1855ab51b369","Type":"ContainerStarted","Data":"58c975ec962419897dbf7437db78777f69178bec2240144a71e08d3e3e4e0995"}
Sep 15 13:38:57.783584 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:57.783564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44" event={"ID":"34ab498e-e299-4cab-85a5-d295962eadc8","Type":"ContainerStarted","Data":"a0db117400ae618b7c01b54a148096482f4c1651ebcaff34ce84c0d14a149306"}
Sep 15 13:38:58.637031 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:58.636985    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:38:58.637539 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:58.636998    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:38:58.637539 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:58.636985    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:38:58.640459 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:58.640434    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 13:38:58.640592 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:58.640437    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qwvrm\""
Sep 15 13:38:58.640592 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:58.640499    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 13:38:58.640592 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:58.640436    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ztmxf\""
Sep 15 13:38:58.678615 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:58.678579    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:38:58.678755 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:58.678723    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 13:38:58.678755 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:58.678746    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57f75d7c55-whpk5: secret "image-registry-tls" not found
Sep 15 13:38:58.678856 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:58.678813    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls podName:aa7df37b-0f61-4c8e-849c-c680261dc753 nodeName:}" failed. No retries permitted until 2026-09-15 13:39:00.678793421 +0000 UTC m=+38.556696243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls") pod "image-registry-57f75d7c55-whpk5" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753") : secret "image-registry-tls" not found
Sep 15 13:38:58.780047 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:58.780007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kt4ds\" (UID: \"4884128c-7457-4add-85ce-103712920e9b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:38:58.780215 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:58.780084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:38:58.780215 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:58.780150    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 13:38:58.780292 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:58.780217    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert podName:4884128c-7457-4add-85ce-103712920e9b nodeName:}" failed. No retries permitted until 2026-09-15 13:39:00.780199589 +0000 UTC m=+38.658102393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-kt4ds" (UID: "4884128c-7457-4add-85ce-103712920e9b") : secret "networking-console-plugin-cert" not found
Sep 15 13:38:58.780292 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:58.780221    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 13:38:58.780292 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:38:58.780284    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls podName:ea8af82a-61dd-44cb-a4a6-27bba6accd04 nodeName:}" failed. No retries permitted until 2026-09-15 13:39:00.780265736 +0000 UTC m=+38.658168549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls") pod "dns-default-zp7l2" (UID: "ea8af82a-61dd-44cb-a4a6-27bba6accd04") : secret "dns-default-metrics-tls" not found
Sep 15 13:38:59.792578 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:59.792549    2582 generic.go:356] "Generic (PLEG): container finished" podID="e42c6f90-64b3-4a55-bf82-c9ee9a126dc9" containerID="85283d56323700824ddcfe5166ca3aff1053fbba99738f5a540a0df52bbf39a9" exitCode=0
Sep 15 13:38:59.793429 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:38:59.792615    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxgrp" event={"ID":"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9","Type":"ContainerDied","Data":"85283d56323700824ddcfe5166ca3aff1053fbba99738f5a540a0df52bbf39a9"}
Sep 15 13:39:00.698933 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:00.698897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:39:00.699107 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:00.699066    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 13:39:00.699107 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:00.699092    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57f75d7c55-whpk5: secret "image-registry-tls" not found
Sep 15 13:39:00.699263 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:00.699175    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls podName:aa7df37b-0f61-4c8e-849c-c680261dc753 nodeName:}" failed. No retries permitted until 2026-09-15 13:39:04.699153138 +0000 UTC m=+42.577055955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls") pod "image-registry-57f75d7c55-whpk5" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753") : secret "image-registry-tls" not found
Sep 15 13:39:00.799372 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:00.799291    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:39:00.799372 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:00.799317    2582 generic.go:356] "Generic (PLEG): container finished" podID="e42c6f90-64b3-4a55-bf82-c9ee9a126dc9" containerID="fcfeb4c9b597909d38b053f65e1f1f2884088a434c320676b6786d7b1a561b3e" exitCode=0
Sep 15 13:39:00.799826 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:00.799393    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxgrp" event={"ID":"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9","Type":"ContainerDied","Data":"fcfeb4c9b597909d38b053f65e1f1f2884088a434c320676b6786d7b1a561b3e"}
Sep 15 13:39:00.799826 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:00.799399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kt4ds\" (UID: \"4884128c-7457-4add-85ce-103712920e9b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:39:00.799826 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:00.799522    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 13:39:00.799826 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:00.799580    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert podName:4884128c-7457-4add-85ce-103712920e9b nodeName:}" failed. No retries permitted until 2026-09-15 13:39:04.799560655 +0000 UTC m=+42.677463465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-kt4ds" (UID: "4884128c-7457-4add-85ce-103712920e9b") : secret "networking-console-plugin-cert" not found
Sep 15 13:39:00.799826 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:00.799612    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 13:39:00.799826 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:00.799677    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls podName:ea8af82a-61dd-44cb-a4a6-27bba6accd04 nodeName:}" failed. No retries permitted until 2026-09-15 13:39:04.799659647 +0000 UTC m=+42.677562461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls") pod "dns-default-zp7l2" (UID: "ea8af82a-61dd-44cb-a4a6-27bba6accd04") : secret "dns-default-metrics-tls" not found
Sep 15 13:39:01.039640 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:01.039581    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:01.436314 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:01.436271    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:01.538033 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:01.537992    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:01.810499 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:01.810409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxgrp" event={"ID":"e42c6f90-64b3-4a55-bf82-c9ee9a126dc9","Type":"ContainerStarted","Data":"5db600e24d156b808be0f768a3a5fe76bde0f6c976865a4aa69507f4cbdd5791"}
Sep 15 13:39:01.833376 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:01.833300    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dxgrp" podStartSLOduration=8.589320584 podStartE2EDuration="39.833279881s" totalImagesPullingTime="31.243959297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:38:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:01.833071008 +0000 UTC m=+39.710973826" watchObservedRunningTime="2026-09-15 13:39:01.833279881 +0000 UTC m=+39.711182698"
Sep 15 13:39:01.952382 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:01.952350    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:02.098825 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:02.098762    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:03.701450 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:03.701420    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:03.753200 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:03.753165    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:04.731987 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:04.731948    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:39:04.732481 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:04.732116    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 13:39:04.732481 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:04.732153    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57f75d7c55-whpk5: secret "image-registry-tls" not found
Sep 15 13:39:04.732481 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:04.732225    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls podName:aa7df37b-0f61-4c8e-849c-c680261dc753 nodeName:}" failed. No retries permitted until 2026-09-15 13:39:12.732205495 +0000 UTC m=+50.610108330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls") pod "image-registry-57f75d7c55-whpk5" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753") : secret "image-registry-tls" not found
Sep 15 13:39:04.820086 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:04.820051    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-xwgbl" event={"ID":"d6d042ce-39af-404c-865d-440addce75f9","Type":"ContainerStarted","Data":"64871d6d1f9c2e2e5dae8ca81b394ab6c1badc56b574093775be85f80cd9203b"}
Sep 15 13:39:04.821763 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:04.821735    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cqqkh" event={"ID":"aaa9a50a-44c0-4d90-9a3b-1855ab51b369","Type":"ContainerStarted","Data":"32d84dc24c94c36ed730bba387954dd43a69972415472ebb497fae09f1e52e36"}
Sep 15 13:39:04.823343 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:04.823319    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44" event={"ID":"34ab498e-e299-4cab-85a5-d295962eadc8","Type":"ContainerStarted","Data":"91d0599cf484c858bcf8c9aa988471e641bf9b4e1b2fd3ee9498e1c55cb374c8"}
Sep 15 13:39:04.837343 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:04.837274    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kt4ds\" (UID: \"4884128c-7457-4add-85ce-103712920e9b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:39:04.837483 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:04.837464    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 13:39:04.837541 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:04.837472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:39:04.837541 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:04.837536    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert podName:4884128c-7457-4add-85ce-103712920e9b nodeName:}" failed. No retries permitted until 2026-09-15 13:39:12.837514211 +0000 UTC m=+50.715417008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-kt4ds" (UID: "4884128c-7457-4add-85ce-103712920e9b") : secret "networking-console-plugin-cert" not found
Sep 15 13:39:04.837650 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:04.837562    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 13:39:04.837650 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:04.837628    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls podName:ea8af82a-61dd-44cb-a4a6-27bba6accd04 nodeName:}" failed. No retries permitted until 2026-09-15 13:39:12.837592833 +0000 UTC m=+50.715495647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls") pod "dns-default-zp7l2" (UID: "ea8af82a-61dd-44cb-a4a6-27bba6accd04") : secret "dns-default-metrics-tls" not found
Sep 15 13:39:04.837650 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:04.837608    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-xwgbl" podStartSLOduration=35.397630162 podStartE2EDuration="41.837579236s" totalImagesPullingTime="6.439949074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:04.837194591 +0000 UTC m=+42.715097411" watchObservedRunningTime="2026-09-15 13:39:04.837579236 +0000 UTC m=+42.715482100"
Sep 15 13:39:04.855568 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:04.855516    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cqqkh" podStartSLOduration=2.444817347 podStartE2EDuration="8.855497642s" totalImagesPullingTime="6.410680295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:04.854729507 +0000 UTC m=+42.732632325" watchObservedRunningTime="2026-09-15 13:39:04.855497642 +0000 UTC m=+42.733400466"
Sep 15 13:39:04.869947 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:04.869901    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44" podStartSLOduration=35.401995238 podStartE2EDuration="41.869885852s" totalImagesPullingTime="6.467890614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:04.869085828 +0000 UTC m=+42.746988649" watchObservedRunningTime="2026-09-15 13:39:04.869885852 +0000 UTC m=+42.747788661"
Sep 15 13:39:05.678840 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:05.678805    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:05.744334 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:05.744302    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:06.100603 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:06.100570    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:06.201771 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:06.201740    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:06.380955 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:06.380887    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:06.457514 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:06.457482    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:06.701830 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:06.701758    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:07.971525 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:07.971495    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rkng4"]
Sep 15 13:39:07.989387 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:07.989352    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:07.997804 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:07.997785    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rkng4"]
Sep 15 13:39:07.997881 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:07.997871    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-rkng4"
Sep 15 13:39:08.000350 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.000330    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 15 13:39:08.001264 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.001247    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ch4gp\""
Sep 15 13:39:08.001345 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.001300    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 15 13:39:08.001345 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.001310    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 15 13:39:08.001453 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.001306    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 15 13:39:08.055216 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.055179    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:08.164555 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.164518    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvs5x\" (UniqueName: \"kubernetes.io/projected/40c2c5bd-fe06-4df4-89f1-59acb1e5241b-kube-api-access-jvs5x\") pod \"service-ca-568869fdc5-rkng4\" (UID: \"40c2c5bd-fe06-4df4-89f1-59acb1e5241b\") " pod="openshift-service-ca/service-ca-568869fdc5-rkng4"
Sep 15 13:39:08.164720 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.164657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/40c2c5bd-fe06-4df4-89f1-59acb1e5241b-signing-key\") pod \"service-ca-568869fdc5-rkng4\" (UID: \"40c2c5bd-fe06-4df4-89f1-59acb1e5241b\") " pod="openshift-service-ca/service-ca-568869fdc5-rkng4"
Sep 15 13:39:08.164720 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.164687    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/40c2c5bd-fe06-4df4-89f1-59acb1e5241b-signing-cabundle\") pod \"service-ca-568869fdc5-rkng4\" (UID: \"40c2c5bd-fe06-4df4-89f1-59acb1e5241b\") " pod="openshift-service-ca/service-ca-568869fdc5-rkng4"
Sep 15 13:39:08.265223 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.265119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/40c2c5bd-fe06-4df4-89f1-59acb1e5241b-signing-key\") pod \"service-ca-568869fdc5-rkng4\" (UID: \"40c2c5bd-fe06-4df4-89f1-59acb1e5241b\") " pod="openshift-service-ca/service-ca-568869fdc5-rkng4"
Sep 15 13:39:08.265223 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.265177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/40c2c5bd-fe06-4df4-89f1-59acb1e5241b-signing-cabundle\") pod \"service-ca-568869fdc5-rkng4\" (UID: \"40c2c5bd-fe06-4df4-89f1-59acb1e5241b\") " pod="openshift-service-ca/service-ca-568869fdc5-rkng4"
Sep 15 13:39:08.265223 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.265217    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvs5x\" (UniqueName: \"kubernetes.io/projected/40c2c5bd-fe06-4df4-89f1-59acb1e5241b-kube-api-access-jvs5x\") pod \"service-ca-568869fdc5-rkng4\" (UID: \"40c2c5bd-fe06-4df4-89f1-59acb1e5241b\") " pod="openshift-service-ca/service-ca-568869fdc5-rkng4"
Sep 15 13:39:08.266022 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.266000    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/40c2c5bd-fe06-4df4-89f1-59acb1e5241b-signing-cabundle\") pod \"service-ca-568869fdc5-rkng4\" (UID: \"40c2c5bd-fe06-4df4-89f1-59acb1e5241b\") " pod="openshift-service-ca/service-ca-568869fdc5-rkng4"
Sep 15 13:39:08.266783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.266752    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:08.268805 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.268787    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/40c2c5bd-fe06-4df4-89f1-59acb1e5241b-signing-key\") pod \"service-ca-568869fdc5-rkng4\" (UID: \"40c2c5bd-fe06-4df4-89f1-59acb1e5241b\") " pod="openshift-service-ca/service-ca-568869fdc5-rkng4"
Sep 15 13:39:08.274182 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.274153    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvs5x\" (UniqueName: \"kubernetes.io/projected/40c2c5bd-fe06-4df4-89f1-59acb1e5241b-kube-api-access-jvs5x\") pod \"service-ca-568869fdc5-rkng4\" (UID: \"40c2c5bd-fe06-4df4-89f1-59acb1e5241b\") " pod="openshift-service-ca/service-ca-568869fdc5-rkng4"
Sep 15 13:39:08.306912 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.306886    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-rkng4"
Sep 15 13:39:08.366565 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.366524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:08.610172 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.610125    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rkng4"]
Sep 15 13:39:08.613408 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:39:08.613379    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c2c5bd_fe06_4df4_89f1_59acb1e5241b.slice/crio-1baf53ac582c99d94a7b85ec64e09af85ad15f871912b5ea25596baeaca93286 WatchSource:0}: Error finding container 1baf53ac582c99d94a7b85ec64e09af85ad15f871912b5ea25596baeaca93286: Status 404 returned error can't find the container with id 1baf53ac582c99d94a7b85ec64e09af85ad15f871912b5ea25596baeaca93286
Sep 15 13:39:08.834781 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.834742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-rkng4" event={"ID":"40c2c5bd-fe06-4df4-89f1-59acb1e5241b","Type":"ContainerStarted","Data":"3a2eb2ce89d98455200ad7a24b98cc86dbac8b782e06e97b9c8e594be408d75c"}
Sep 15 13:39:08.834977 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.834797    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-rkng4" event={"ID":"40c2c5bd-fe06-4df4-89f1-59acb1e5241b","Type":"ContainerStarted","Data":"1baf53ac582c99d94a7b85ec64e09af85ad15f871912b5ea25596baeaca93286"}
Sep 15 13:39:08.852889 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.852835    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-rkng4" podStartSLOduration=1.852820761 podStartE2EDuration="1.852820761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:08.851778718 +0000 UTC m=+46.729681548" watchObservedRunningTime="2026-09-15 13:39:08.852820761 +0000 UTC m=+46.730723577"
Sep 15 13:39:08.956205 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:08.956119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:10.214231 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:10.214193    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:10.510604 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:10.510479    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:10.600533 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:10.600495    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:10.700562 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:10.700514    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:10.904680 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:10.904641    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:11.929348 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:11.929313    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:12.303907 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:12.303834    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:12.504940 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:12.504902    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:12.712066 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:12.712032    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:12.802033 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:12.801994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:39:12.802225 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:12.802134    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 13:39:12.802225 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:12.802167    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57f75d7c55-whpk5: secret "image-registry-tls" not found
Sep 15 13:39:12.802225 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:12.802221    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls podName:aa7df37b-0f61-4c8e-849c-c680261dc753 nodeName:}" failed. No retries permitted until 2026-09-15 13:39:28.802205919 +0000 UTC m=+66.680108716 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls") pod "image-registry-57f75d7c55-whpk5" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753") : secret "image-registry-tls" not found
Sep 15 13:39:12.902659 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:12.902624    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:39:12.902801 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:12.902701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kt4ds\" (UID: \"4884128c-7457-4add-85ce-103712920e9b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:39:12.902801 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:12.902776    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 13:39:12.902886 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:12.902841    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls podName:ea8af82a-61dd-44cb-a4a6-27bba6accd04 nodeName:}" failed. No retries permitted until 2026-09-15 13:39:28.902821185 +0000 UTC m=+66.780723999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls") pod "dns-default-zp7l2" (UID: "ea8af82a-61dd-44cb-a4a6-27bba6accd04") : secret "dns-default-metrics-tls" not found
Sep 15 13:39:12.902886 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:12.902782    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 13:39:12.902886 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:12.902877    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert podName:4884128c-7457-4add-85ce-103712920e9b nodeName:}" failed. No retries permitted until 2026-09-15 13:39:28.902870854 +0000 UTC m=+66.780773652 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-kt4ds" (UID: "4884128c-7457-4add-85ce-103712920e9b") : secret "networking-console-plugin-cert" not found
Sep 15 13:39:13.113357 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:13.113328    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:13.229663 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:13.229634    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:13.329475 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:13.329437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:14.733359 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:14.733323    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:14.832835 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:14.832800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:15.300949 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:15.300915    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:17.217780 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:17.217751    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:17.272823 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:17.272796    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:17.374234 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:17.374199    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:17.899834 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:17.899800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:18.104009 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:18.103974    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:18.300710 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:18.300633    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:18.507418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:18.507390    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:19.558447 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:19.558417    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:19.704285 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:19.704240    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:20.113757 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:20.113716    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:20.251198 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:20.251165    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:20.421056 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:20.420981    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:20.521600 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:20.521546    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:22.775904 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:22.775866    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttnsl"
Sep 15 13:39:28.325660 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.325615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdmc\" (UniqueName: \"kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc\") pod \"network-check-target-j5pmd\" (UID: \"ef84ca8e-d344-4d38-b015-8a76a03e5dfa\") " pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:39:28.328114 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.328094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbdmc\" (UniqueName: \"kubernetes.io/projected/ef84ca8e-d344-4d38-b015-8a76a03e5dfa-kube-api-access-sbdmc\") pod \"network-check-target-j5pmd\" (UID: \"ef84ca8e-d344-4d38-b015-8a76a03e5dfa\") " pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:39:28.360830 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.360798    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ztmxf\""
Sep 15 13:39:28.368956 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.368932    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:39:28.426684 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.426646    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs\") pod \"network-metrics-daemon-5kxxr\" (UID: \"0258dc54-5388-49fa-bd24-0ee885930283\") " pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:39:28.426814 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.426698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:39:28.431464 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.431431    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 13:39:28.431581 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.431515    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 13:39:28.439786 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.439761    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe-original-pull-secret\") pod \"global-pull-secret-syncer-fbj84\" (UID: \"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe\") " pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:39:28.439962 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.439945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0258dc54-5388-49fa-bd24-0ee885930283-metrics-certs\") pod \"network-metrics-daemon-5kxxr\" (UID: \"0258dc54-5388-49fa-bd24-0ee885930283\") " pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:39:28.639715 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:39:28.639686    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef84ca8e_d344_4d38_b015_8a76a03e5dfa.slice/crio-557c3373223b48018f13477d8ce21e1db8a8c85c74c7cac5f332048c7ea68d13 WatchSource:0}: Error finding container 557c3373223b48018f13477d8ce21e1db8a8c85c74c7cac5f332048c7ea68d13: Status 404 returned error can't find the container with id 557c3373223b48018f13477d8ce21e1db8a8c85c74c7cac5f332048c7ea68d13
Sep 15 13:39:28.640217 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.640200    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5pmd"]
Sep 15 13:39:28.649896 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.649874    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fbj84"
Sep 15 13:39:28.666983 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.666959    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qwvrm\""
Sep 15 13:39:28.675080 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.675059    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kxxr"
Sep 15 13:39:28.830776 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.830733    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:39:28.833122 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.833092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls\") pod \"image-registry-57f75d7c55-whpk5\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") " pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:39:28.886674 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.886643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5pmd" event={"ID":"ef84ca8e-d344-4d38-b015-8a76a03e5dfa","Type":"ContainerStarted","Data":"65b4d28e8e53354ec4b3d8c349fafd21e544aca3cd5c52c7810e9ab68dfd69e2"}
Sep 15 13:39:28.886799 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.886690    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5pmd" event={"ID":"ef84ca8e-d344-4d38-b015-8a76a03e5dfa","Type":"ContainerStarted","Data":"557c3373223b48018f13477d8ce21e1db8a8c85c74c7cac5f332048c7ea68d13"}
Sep 15 13:39:28.886799 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.886761    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:39:28.906681 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.906587    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j5pmd" podStartSLOduration=66.906569256 podStartE2EDuration="1m6.906569256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:28.906415281 +0000 UTC m=+66.784318101" watchObservedRunningTime="2026-09-15 13:39:28.906569256 +0000 UTC m=+66.784472079"
Sep 15 13:39:28.920221 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.920194    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fbj84"]
Sep 15 13:39:28.922988 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:39:28.922956    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b844a8_6a0e_4d28_8501_7d96e3cb6dfe.slice/crio-90c0b5c5123506b83a453a709c38e5a50e91a00e902b14c06d79d05feb0d0476 WatchSource:0}: Error finding container 90c0b5c5123506b83a453a709c38e5a50e91a00e902b14c06d79d05feb0d0476: Status 404 returned error can't find the container with id 90c0b5c5123506b83a453a709c38e5a50e91a00e902b14c06d79d05feb0d0476
Sep 15 13:39:28.931594 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.931568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kt4ds\" (UID: \"4884128c-7457-4add-85ce-103712920e9b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:39:28.931682 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.931625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:39:28.933891 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.933871    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4884128c-7457-4add-85ce-103712920e9b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kt4ds\" (UID: \"4884128c-7457-4add-85ce-103712920e9b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:39:28.933891 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.933884    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea8af82a-61dd-44cb-a4a6-27bba6accd04-metrics-tls\") pod \"dns-default-zp7l2\" (UID: \"ea8af82a-61dd-44cb-a4a6-27bba6accd04\") " pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:39:28.944016 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:28.943996    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5kxxr"]
Sep 15 13:39:28.946929 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:39:28.946908    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0258dc54_5388_49fa_bd24_0ee885930283.slice/crio-89f6e9b45f281798bb6b695f423caf9a2b4cd99092b5554b33a50ab2cdfa5ae2 WatchSource:0}: Error finding container 89f6e9b45f281798bb6b695f423caf9a2b4cd99092b5554b33a50ab2cdfa5ae2: Status 404 returned error can't find the container with id 89f6e9b45f281798bb6b695f423caf9a2b4cd99092b5554b33a50ab2cdfa5ae2
Sep 15 13:39:29.122369 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.122341    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kxsf6\""
Sep 15 13:39:29.130151 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.130121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:39:29.148275 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.148254    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dfkpb\""
Sep 15 13:39:29.156451 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.156426    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:39:29.165775 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.165745    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-r9fql\""
Sep 15 13:39:29.173986 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.173965    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"
Sep 15 13:39:29.406910 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.406645    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57f75d7c55-whpk5"]
Sep 15 13:39:29.409537 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:39:29.409469    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7df37b_0f61_4c8e_849c_c680261dc753.slice/crio-f0f3e596901e56fc0372c32b38d11d5a6a1cdc7e01a4821a60436617ac42de0c WatchSource:0}: Error finding container f0f3e596901e56fc0372c32b38d11d5a6a1cdc7e01a4821a60436617ac42de0c: Status 404 returned error can't find the container with id f0f3e596901e56fc0372c32b38d11d5a6a1cdc7e01a4821a60436617ac42de0c
Sep 15 13:39:29.437719 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.437690    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zp7l2"]
Sep 15 13:39:29.441711 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:39:29.441679    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8af82a_61dd_44cb_a4a6_27bba6accd04.slice/crio-3a4bbe1ece0b044cdd62aff884ff8db3f7a13f4595e09f57d7cf4adc91a5dc5d WatchSource:0}: Error finding container 3a4bbe1ece0b044cdd62aff884ff8db3f7a13f4595e09f57d7cf4adc91a5dc5d: Status 404 returned error can't find the container with id 3a4bbe1ece0b044cdd62aff884ff8db3f7a13f4595e09f57d7cf4adc91a5dc5d
Sep 15 13:39:29.457077 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.457048    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds"]
Sep 15 13:39:29.460682 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:39:29.460653    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4884128c_7457_4add_85ce_103712920e9b.slice/crio-99a3282bac03ebbb37b17b591652292167e2091d30e6ded8ad7ddf6894cd0c7c WatchSource:0}: Error finding container 99a3282bac03ebbb37b17b591652292167e2091d30e6ded8ad7ddf6894cd0c7c: Status 404 returned error can't find the container with id 99a3282bac03ebbb37b17b591652292167e2091d30e6ded8ad7ddf6894cd0c7c
Sep 15 13:39:29.892695 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.892659    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zp7l2" event={"ID":"ea8af82a-61dd-44cb-a4a6-27bba6accd04","Type":"ContainerStarted","Data":"3a4bbe1ece0b044cdd62aff884ff8db3f7a13f4595e09f57d7cf4adc91a5dc5d"}
Sep 15 13:39:29.894097 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.894064    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds" event={"ID":"4884128c-7457-4add-85ce-103712920e9b","Type":"ContainerStarted","Data":"99a3282bac03ebbb37b17b591652292167e2091d30e6ded8ad7ddf6894cd0c7c"}
Sep 15 13:39:29.900104 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.900072    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57f75d7c55-whpk5" event={"ID":"aa7df37b-0f61-4c8e-849c-c680261dc753","Type":"ContainerStarted","Data":"f295f0e38947eefa4efd171e0c3e76d74345747d791c08ebd571cad150a3c951"}
Sep 15 13:39:29.900247 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.900124    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57f75d7c55-whpk5" event={"ID":"aa7df37b-0f61-4c8e-849c-c680261dc753","Type":"ContainerStarted","Data":"f0f3e596901e56fc0372c32b38d11d5a6a1cdc7e01a4821a60436617ac42de0c"}
Sep 15 13:39:29.900377 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.900350    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:39:29.902274 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.902246    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kxxr" event={"ID":"0258dc54-5388-49fa-bd24-0ee885930283","Type":"ContainerStarted","Data":"89f6e9b45f281798bb6b695f423caf9a2b4cd99092b5554b33a50ab2cdfa5ae2"}
Sep 15 13:39:29.903588 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.903560    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fbj84" event={"ID":"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe","Type":"ContainerStarted","Data":"90c0b5c5123506b83a453a709c38e5a50e91a00e902b14c06d79d05feb0d0476"}
Sep 15 13:39:29.921855 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:29.921800    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57f75d7c55-whpk5" podStartSLOduration=66.921783313 podStartE2EDuration="1m6.921783313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:29.920295948 +0000 UTC m=+67.798198769" watchObservedRunningTime="2026-09-15 13:39:29.921783313 +0000 UTC m=+67.799686131"
Sep 15 13:39:30.910591 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:30.910551    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kxxr" event={"ID":"0258dc54-5388-49fa-bd24-0ee885930283","Type":"ContainerStarted","Data":"586b64fe551cdb53c5a682506430bb84848eefd0054e89440685df2d5a50d67c"}
Sep 15 13:39:30.911036 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:30.910623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kxxr" event={"ID":"0258dc54-5388-49fa-bd24-0ee885930283","Type":"ContainerStarted","Data":"dfd37b4d7e4264bac760550a8a4e588e4c402107ea322ca1e5a1d584314bc65a"}
Sep 15 13:39:30.912449 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:30.912411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds" event={"ID":"4884128c-7457-4add-85ce-103712920e9b","Type":"ContainerStarted","Data":"fba7394593006979ccaa12ec2bcf4c4ab0b59fac896209a5cec8e3dc35ee34e5"}
Sep 15 13:39:30.927704 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:30.927641    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5kxxr" podStartSLOduration=67.965516697 podStartE2EDuration="1m8.927626827s" totalImagesPullingTime="962.11013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:30.926302394 +0000 UTC m=+68.804205214" watchObservedRunningTime="2026-09-15 13:39:30.927626827 +0000 UTC m=+68.805529645"
Sep 15 13:39:30.943757 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:30.941248    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kt4ds" podStartSLOduration=40.671054941 podStartE2EDuration="41.94122948s" totalImagesPullingTime="1.270174539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:30.940659439 +0000 UTC m=+68.818562259" watchObservedRunningTime="2026-09-15 13:39:30.94122948 +0000 UTC m=+68.819132300"
Sep 15 13:39:32.630663 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.630626    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-gl2rg"]
Sep 15 13:39:32.648084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.648015    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"]
Sep 15 13:39:32.648351 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.648322    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.651327 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.651303    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gvdsb\""
Sep 15 13:39:32.652020 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.652000    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 15 13:39:32.652850 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.652612    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 15 13:39:32.652850 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.652638    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 15 13:39:32.653063 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.652981    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 15 13:39:32.661735 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.661682    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 15 13:39:32.661996 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.661973    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"]
Sep 15 13:39:32.662088 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.662017    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-gl2rg"]
Sep 15 13:39:32.662492 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.662473    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"
Sep 15 13:39:32.664686 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.664645    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 13:39:32.665855 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.665645    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 13:39:32.665855 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.665674    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 15 13:39:32.666641 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.666487    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 13:39:32.667738 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.667720    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"]
Sep 15 13:39:32.688210 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.688180    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"]
Sep 15 13:39:32.688370 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.688246    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.690475 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.690454    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 15 13:39:32.690617 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.690584    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 15 13:39:32.690782 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.690761    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 15 13:39:32.690869 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.690851    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 15 13:39:32.714062 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.714028    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:32.759021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.758988    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjwt\" (UniqueName: \"kubernetes.io/projected/519921d5-6953-48a0-92e3-1595172bd6ab-kube-api-access-nbjwt\") pod \"console-operator-696884fb9c-gl2rg\" (UID: \"519921d5-6953-48a0-92e3-1595172bd6ab\") " pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.759021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.759035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38c59113-4fcd-4088-b564-42dd58dbecab-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.759262 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.759070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38c59113-4fcd-4088-b564-42dd58dbecab-ca\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.759262 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.759093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb9a3ed2-a613-4455-82e7-d39e9ba4880b-tmp\") pod \"klusterlet-addon-workmgr-b998cb858-jdf9j\" (UID: \"eb9a3ed2-a613-4455-82e7-d39e9ba4880b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"
Sep 15 13:39:32.759262 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.759110    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38c59113-4fcd-4088-b564-42dd58dbecab-hub\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.759262 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.759123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/519921d5-6953-48a0-92e3-1595172bd6ab-trusted-ca\") pod \"console-operator-696884fb9c-gl2rg\" (UID: \"519921d5-6953-48a0-92e3-1595172bd6ab\") " pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.759262 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.759166    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eb9a3ed2-a613-4455-82e7-d39e9ba4880b-klusterlet-config\") pod \"klusterlet-addon-workmgr-b998cb858-jdf9j\" (UID: \"eb9a3ed2-a613-4455-82e7-d39e9ba4880b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"
Sep 15 13:39:32.759497 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.759272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9v6\" (UniqueName: \"kubernetes.io/projected/38c59113-4fcd-4088-b564-42dd58dbecab-kube-api-access-nw9v6\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.759497 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.759294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl6mr\" (UniqueName: \"kubernetes.io/projected/eb9a3ed2-a613-4455-82e7-d39e9ba4880b-kube-api-access-tl6mr\") pod \"klusterlet-addon-workmgr-b998cb858-jdf9j\" (UID: \"eb9a3ed2-a613-4455-82e7-d39e9ba4880b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"
Sep 15 13:39:32.759497 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.759321    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c59113-4fcd-4088-b564-42dd58dbecab-additional-service-ca\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.759497 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.759346    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/519921d5-6953-48a0-92e3-1595172bd6ab-config\") pod \"console-operator-696884fb9c-gl2rg\" (UID: \"519921d5-6953-48a0-92e3-1595172bd6ab\") " pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.759497 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.759394    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38c59113-4fcd-4088-b564-42dd58dbecab-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.759497 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.759435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/519921d5-6953-48a0-92e3-1595172bd6ab-serving-cert\") pod \"console-operator-696884fb9c-gl2rg\" (UID: \"519921d5-6953-48a0-92e3-1595172bd6ab\") " pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.860403 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38c59113-4fcd-4088-b564-42dd58dbecab-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.860571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38c59113-4fcd-4088-b564-42dd58dbecab-ca\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.860571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860554    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb9a3ed2-a613-4455-82e7-d39e9ba4880b-tmp\") pod \"klusterlet-addon-workmgr-b998cb858-jdf9j\" (UID: \"eb9a3ed2-a613-4455-82e7-d39e9ba4880b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"
Sep 15 13:39:32.860670 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38c59113-4fcd-4088-b564-42dd58dbecab-hub\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.860670 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860616    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/519921d5-6953-48a0-92e3-1595172bd6ab-trusted-ca\") pod \"console-operator-696884fb9c-gl2rg\" (UID: \"519921d5-6953-48a0-92e3-1595172bd6ab\") " pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.860670 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eb9a3ed2-a613-4455-82e7-d39e9ba4880b-klusterlet-config\") pod \"klusterlet-addon-workmgr-b998cb858-jdf9j\" (UID: \"eb9a3ed2-a613-4455-82e7-d39e9ba4880b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"
Sep 15 13:39:32.860818 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9v6\" (UniqueName: \"kubernetes.io/projected/38c59113-4fcd-4088-b564-42dd58dbecab-kube-api-access-nw9v6\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.860818 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860721    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl6mr\" (UniqueName: \"kubernetes.io/projected/eb9a3ed2-a613-4455-82e7-d39e9ba4880b-kube-api-access-tl6mr\") pod \"klusterlet-addon-workmgr-b998cb858-jdf9j\" (UID: \"eb9a3ed2-a613-4455-82e7-d39e9ba4880b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"
Sep 15 13:39:32.860818 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c59113-4fcd-4088-b564-42dd58dbecab-additional-service-ca\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.860818 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/519921d5-6953-48a0-92e3-1595172bd6ab-config\") pod \"console-operator-696884fb9c-gl2rg\" (UID: \"519921d5-6953-48a0-92e3-1595172bd6ab\") " pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.860818 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860814    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38c59113-4fcd-4088-b564-42dd58dbecab-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.861062 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/519921d5-6953-48a0-92e3-1595172bd6ab-serving-cert\") pod \"console-operator-696884fb9c-gl2rg\" (UID: \"519921d5-6953-48a0-92e3-1595172bd6ab\") " pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.861062 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860887    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjwt\" (UniqueName: \"kubernetes.io/projected/519921d5-6953-48a0-92e3-1595172bd6ab-kube-api-access-nbjwt\") pod \"console-operator-696884fb9c-gl2rg\" (UID: \"519921d5-6953-48a0-92e3-1595172bd6ab\") " pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.861062 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.860965    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb9a3ed2-a613-4455-82e7-d39e9ba4880b-tmp\") pod \"klusterlet-addon-workmgr-b998cb858-jdf9j\" (UID: \"eb9a3ed2-a613-4455-82e7-d39e9ba4880b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"
Sep 15 13:39:32.861826 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.861561    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/519921d5-6953-48a0-92e3-1595172bd6ab-trusted-ca\") pod \"console-operator-696884fb9c-gl2rg\" (UID: \"519921d5-6953-48a0-92e3-1595172bd6ab\") " pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.861826 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.861696    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/519921d5-6953-48a0-92e3-1595172bd6ab-config\") pod \"console-operator-696884fb9c-gl2rg\" (UID: \"519921d5-6953-48a0-92e3-1595172bd6ab\") " pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.863630 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.863582    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38c59113-4fcd-4088-b564-42dd58dbecab-hub\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.863792 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.862823    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c59113-4fcd-4088-b564-42dd58dbecab-additional-service-ca\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.863855 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.863803    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38c59113-4fcd-4088-b564-42dd58dbecab-ca\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.864761 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.864729    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/519921d5-6953-48a0-92e3-1595172bd6ab-serving-cert\") pod \"console-operator-696884fb9c-gl2rg\" (UID: \"519921d5-6953-48a0-92e3-1595172bd6ab\") " pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.865568 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.865540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38c59113-4fcd-4088-b564-42dd58dbecab-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.865855 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.865831    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eb9a3ed2-a613-4455-82e7-d39e9ba4880b-klusterlet-config\") pod \"klusterlet-addon-workmgr-b998cb858-jdf9j\" (UID: \"eb9a3ed2-a613-4455-82e7-d39e9ba4880b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"
Sep 15 13:39:32.865952 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.865841    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38c59113-4fcd-4088-b564-42dd58dbecab-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.877751 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.877726    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjwt\" (UniqueName: \"kubernetes.io/projected/519921d5-6953-48a0-92e3-1595172bd6ab-kube-api-access-nbjwt\") pod \"console-operator-696884fb9c-gl2rg\" (UID: \"519921d5-6953-48a0-92e3-1595172bd6ab\") " pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.879345 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.879295    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9v6\" (UniqueName: \"kubernetes.io/projected/38c59113-4fcd-4088-b564-42dd58dbecab-kube-api-access-nw9v6\") pod \"cluster-proxy-proxy-agent-967d7448-qpdgf\" (UID: \"38c59113-4fcd-4088-b564-42dd58dbecab\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:32.879551 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.879523    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:32.881204 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.881129    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl6mr\" (UniqueName: \"kubernetes.io/projected/eb9a3ed2-a613-4455-82e7-d39e9ba4880b-kube-api-access-tl6mr\") pod \"klusterlet-addon-workmgr-b998cb858-jdf9j\" (UID: \"eb9a3ed2-a613-4455-82e7-d39e9ba4880b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"
Sep 15 13:39:32.960922 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.960887    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:32.974312 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.974285    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"
Sep 15 13:39:32.980270 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:32.980240    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:33.005311 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:33.005194    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"
Sep 15 13:39:33.082301 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:33.082258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:33.181024 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:33.180926    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:33.707818 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:33.707770    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:33.765625 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:33.765550    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:34.286542 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:34.286511    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-gl2rg"]
Sep 15 13:39:34.292213 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:39:34.292177    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519921d5_6953_48a0_92e3_1595172bd6ab.slice/crio-3588cf137bffa3e620bded4af865a70a2ec4dc9965f665041440c6b164eb7beb WatchSource:0}: Error finding container 3588cf137bffa3e620bded4af865a70a2ec4dc9965f665041440c6b164eb7beb: Status 404 returned error can't find the container with id 3588cf137bffa3e620bded4af865a70a2ec4dc9965f665041440c6b164eb7beb
Sep 15 13:39:34.323544 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:34.323517    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"]
Sep 15 13:39:34.327031 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:39:34.327003    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9a3ed2_a613_4455_82e7_d39e9ba4880b.slice/crio-5d3c6237f7396d152d6bb9fd8a9ee6ffebbb18552a3b5225d7cffbcd18431e2a WatchSource:0}: Error finding container 5d3c6237f7396d152d6bb9fd8a9ee6ffebbb18552a3b5225d7cffbcd18431e2a: Status 404 returned error can't find the container with id 5d3c6237f7396d152d6bb9fd8a9ee6ffebbb18552a3b5225d7cffbcd18431e2a
Sep 15 13:39:34.340026 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:34.339972    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf"]
Sep 15 13:39:34.346840 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:39:34.346812    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38c59113_4fcd_4088_b564_42dd58dbecab.slice/crio-e3698549b1dfbbc0b883e16668492964292c1f1d8f36027686de6fe5bd8ff82c WatchSource:0}: Error finding container e3698549b1dfbbc0b883e16668492964292c1f1d8f36027686de6fe5bd8ff82c: Status 404 returned error can't find the container with id e3698549b1dfbbc0b883e16668492964292c1f1d8f36027686de6fe5bd8ff82c
Sep 15 13:39:34.929644 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:34.929603    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zp7l2" event={"ID":"ea8af82a-61dd-44cb-a4a6-27bba6accd04","Type":"ContainerStarted","Data":"06a1fead232a59369bf76ff0c318ecd13c8af3d696eb35df2f78a664fe528b65"}
Sep 15 13:39:34.930154 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:34.929664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zp7l2" event={"ID":"ea8af82a-61dd-44cb-a4a6-27bba6accd04","Type":"ContainerStarted","Data":"6e3ae54e85fccc9d31e0f4050b54bc4046acc8703f7678b4a28d22ad28341c35"}
Sep 15 13:39:34.930154 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:34.929718    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:39:34.931176 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:34.931125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf" event={"ID":"38c59113-4fcd-4088-b564-42dd58dbecab","Type":"ContainerStarted","Data":"e3698549b1dfbbc0b883e16668492964292c1f1d8f36027686de6fe5bd8ff82c"}
Sep 15 13:39:34.932452 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:34.932422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-gl2rg" event={"ID":"519921d5-6953-48a0-92e3-1595172bd6ab","Type":"ContainerStarted","Data":"3588cf137bffa3e620bded4af865a70a2ec4dc9965f665041440c6b164eb7beb"}
Sep 15 13:39:34.934265 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:34.934235    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fbj84" event={"ID":"b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe","Type":"ContainerStarted","Data":"44b6cb7b77af4449ede4aa2019207f5a322bbe5fbc47a3283ced978a6f32719c"}
Sep 15 13:39:34.935622 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:34.935588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j" event={"ID":"eb9a3ed2-a613-4455-82e7-d39e9ba4880b","Type":"ContainerStarted","Data":"5d3c6237f7396d152d6bb9fd8a9ee6ffebbb18552a3b5225d7cffbcd18431e2a"}
Sep 15 13:39:34.948457 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:34.948401    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zp7l2" podStartSLOduration=34.385817148 podStartE2EDuration="38.948386639s" totalImagesPullingTime="4.562569491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:34.947865411 +0000 UTC m=+72.825768258" watchObservedRunningTime="2026-09-15 13:39:34.948386639 +0000 UTC m=+72.826289458"
Sep 15 13:39:34.963728 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:34.963670    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fbj84" podStartSLOduration=66.862927726 podStartE2EDuration="1m11.963649501s" totalImagesPullingTime="5.100721775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:34.963405538 +0000 UTC m=+72.841308357" watchObservedRunningTime="2026-09-15 13:39:34.963649501 +0000 UTC m=+72.841552318"
Sep 15 13:39:35.164112 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:35.164039    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:36.134370 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:36.134337    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:37.305531 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.305474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:37.405544 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.405507    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:37.480956 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.480917    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:37.540417 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.540385    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:37.698908 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.698877    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:37.792291 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.792256    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:37.889461 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.889429    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l8mlb"]
Sep 15 13:39:37.891942 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.891918    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:37.894508 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.894471    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 13:39:37.894641 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.894514    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 13:39:37.894641 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.894538    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 13:39:37.895475 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.895360    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 13:39:37.895475 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.895471    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jx2vh\""
Sep 15 13:39:37.902019 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:37.901996    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8mlb"]
Sep 15 13:39:38.016559 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.016462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/163705e9-255b-49b1-93bf-e0a56c9fa48e-crio-socket\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.016559 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.016515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.016786 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.016603    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl68b\" (UniqueName: \"kubernetes.io/projected/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-api-access-nl68b\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.016786 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.016675    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.016904 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.016795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/163705e9-255b-49b1-93bf-e0a56c9fa48e-data-volume\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.117719 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.117679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/163705e9-255b-49b1-93bf-e0a56c9fa48e-data-volume\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.117913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.117732    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/163705e9-255b-49b1-93bf-e0a56c9fa48e-crio-socket\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.117913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.117758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.117913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.117786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl68b\" (UniqueName: \"kubernetes.io/projected/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-api-access-nl68b\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.117913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.117829    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.117913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.117860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/163705e9-255b-49b1-93bf-e0a56c9fa48e-crio-socket\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.118193 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:38.117920    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 13:39:38.118193 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:38.117960    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 13:39:38.118193 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:38.118005    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls podName:163705e9-255b-49b1-93bf-e0a56c9fa48e nodeName:}" failed. No retries permitted until 2026-09-15 13:39:38.617982125 +0000 UTC m=+76.495884939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8mlb" (UID: "163705e9-255b-49b1-93bf-e0a56c9fa48e") : secret "insights-runtime-extractor-tls" not found
Sep 15 13:39:38.118193 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:38.118044    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm podName:163705e9-255b-49b1-93bf-e0a56c9fa48e nodeName:}" failed. No retries permitted until 2026-09-15 13:39:38.618024626 +0000 UTC m=+76.495927442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8mlb" (UID: "163705e9-255b-49b1-93bf-e0a56c9fa48e") : configmap "kube-rbac-proxy" not found
Sep 15 13:39:38.118193 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.118068    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/163705e9-255b-49b1-93bf-e0a56c9fa48e-data-volume\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.126562 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.126537    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl68b\" (UniqueName: \"kubernetes.io/projected/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-api-access-nl68b\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.622670 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.622634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.623082 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.622701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:38.623082 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:38.622778    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 13:39:38.623082 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:38.622793    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 13:39:38.623082 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:38.622865    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm podName:163705e9-255b-49b1-93bf-e0a56c9fa48e nodeName:}" failed. No retries permitted until 2026-09-15 13:39:39.622843659 +0000 UTC m=+77.500746462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8mlb" (UID: "163705e9-255b-49b1-93bf-e0a56c9fa48e") : configmap "kube-rbac-proxy" not found
Sep 15 13:39:38.623082 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:38.622890    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls podName:163705e9-255b-49b1-93bf-e0a56c9fa48e nodeName:}" failed. No retries permitted until 2026-09-15 13:39:39.622879506 +0000 UTC m=+77.500782306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8mlb" (UID: "163705e9-255b-49b1-93bf-e0a56c9fa48e") : secret "insights-runtime-extractor-tls" not found
Sep 15 13:39:38.649996 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.649951    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:38.751230 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:38.751200    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:39.269709 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:39.269659    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:39.343555 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:39.343515    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:39.632833 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:39.632787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:39.633325 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:39.632867    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:39.633325 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:39.632968    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 13:39:39.633325 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:39.633053    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls podName:163705e9-255b-49b1-93bf-e0a56c9fa48e nodeName:}" failed. No retries permitted until 2026-09-15 13:39:41.633023666 +0000 UTC m=+79.510926469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8mlb" (UID: "163705e9-255b-49b1-93bf-e0a56c9fa48e") : secret "insights-runtime-extractor-tls" not found
Sep 15 13:39:39.633325 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:39.632978    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 13:39:39.633325 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:39.633119    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm podName:163705e9-255b-49b1-93bf-e0a56c9fa48e nodeName:}" failed. No retries permitted until 2026-09-15 13:39:41.633098427 +0000 UTC m=+79.511001243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8mlb" (UID: "163705e9-255b-49b1-93bf-e0a56c9fa48e") : configmap "kube-rbac-proxy" not found
Sep 15 13:39:40.065170 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.065064    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zp7l2_ea8af82a-61dd-44cb-a4a6-27bba6accd04/dns/0.log"
Sep 15 13:39:40.264815 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.264778    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zp7l2_ea8af82a-61dd-44cb-a4a6-27bba6accd04/kube-rbac-proxy/0.log"
Sep 15 13:39:40.393616 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.393578    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:40.470049 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.469994    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:40.652563 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.652531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:40.754321 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.754118    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:40.868291 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.868271    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lhd7g_9a1476f2-df89-474d-bbce-2258e2487080/dns-node-resolver/0.log"
Sep 15 13:39:40.963560 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.963480    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf" event={"ID":"38c59113-4fcd-4088-b564-42dd58dbecab","Type":"ContainerStarted","Data":"2f3ea7cc76c740b0baf56be08f2052e30c74397fdfd1f1b1dff716d19faa0b40"}
Sep 15 13:39:40.963560 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.963528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf" event={"ID":"38c59113-4fcd-4088-b564-42dd58dbecab","Type":"ContainerStarted","Data":"8c87def237964e0084bca7a4b1fe49100236512ddec4bbb8e36596ddafb42552"}
Sep 15 13:39:40.963560 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.963548    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf" event={"ID":"38c59113-4fcd-4088-b564-42dd58dbecab","Type":"ContainerStarted","Data":"053b32ccd2c86c8e55c7a331d07e42d0c18ee3ae13438ebdae3b80cd0e082bf9"}
Sep 15 13:39:40.965216 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.965185    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-gl2rg" event={"ID":"519921d5-6953-48a0-92e3-1595172bd6ab","Type":"ContainerStarted","Data":"c86a5118ad48fb788e50ad27e3b9fbe142dfd516652dca38c060f7a770eebb2a"}
Sep 15 13:39:40.965370 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.965351    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:40.966903 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.966879    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j" event={"ID":"eb9a3ed2-a613-4455-82e7-d39e9ba4880b","Type":"ContainerStarted","Data":"c0a83c9c07935fb44f9d29c42b1f6917412adaf9a20fb8f385f643f57a4b28d4"}
Sep 15 13:39:40.967093 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.967073    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"
Sep 15 13:39:40.968559 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.968541    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j"
Sep 15 13:39:40.981728 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.981665    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-967d7448-qpdgf" podStartSLOduration=2.679495004 podStartE2EDuration="8.981650022s" totalImagesPullingTime="6.302155018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:40.980518303 +0000 UTC m=+78.858421124" watchObservedRunningTime="2026-09-15 13:39:40.981650022 +0000 UTC m=+78.859552841"
Sep 15 13:39:40.996403 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:40.996365    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-gl2rg" podStartSLOduration=2.609962716 podStartE2EDuration="8.996352646s" totalImagesPullingTime="6.38638993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:40.995331176 +0000 UTC m=+78.873234008" watchObservedRunningTime="2026-09-15 13:39:40.996352646 +0000 UTC m=+78.874255465"
Sep 15 13:39:41.011058 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:41.011016    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b998cb858-jdf9j" podStartSLOduration=2.688813558 podStartE2EDuration="9.01100364s" totalImagesPullingTime="6.322190082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:41.009922294 +0000 UTC m=+78.887825123" watchObservedRunningTime="2026-09-15 13:39:41.01100364 +0000 UTC m=+78.888906459"
Sep 15 13:39:41.206931 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:41.206901    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-gl2rg"
Sep 15 13:39:41.403419 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:41.403384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:41.465729 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:41.465702    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57f75d7c55-whpk5_aa7df37b-0f61-4c8e-849c-c680261dc753/registry/0.log"
Sep 15 13:39:41.505171 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:41.505132    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:41.581248 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:41.581204    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:41.648693 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:41.648659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:41.648857 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:41.648710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:41.648857 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:41.648801    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 13:39:41.648857 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:41.648803    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 13:39:41.648982 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:41.648865    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls podName:163705e9-255b-49b1-93bf-e0a56c9fa48e nodeName:}" failed. No retries permitted until 2026-09-15 13:39:45.648849801 +0000 UTC m=+83.526752615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8mlb" (UID: "163705e9-255b-49b1-93bf-e0a56c9fa48e") : secret "insights-runtime-extractor-tls" not found
Sep 15 13:39:41.648982 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:41.648882    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm podName:163705e9-255b-49b1-93bf-e0a56c9fa48e nodeName:}" failed. No retries permitted until 2026-09-15 13:39:45.648875227 +0000 UTC m=+83.526778036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8mlb" (UID: "163705e9-255b-49b1-93bf-e0a56c9fa48e") : configmap "kube-rbac-proxy" not found
Sep 15 13:39:41.681174 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:41.681076    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:41.781861 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:41.781824    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:41.867723 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:41.867698    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b7tq6_7edf22ea-7f60-4d71-95db-8594c103b405/node-ca/0.log"
Sep 15 13:39:41.882418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:41.882383    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:41.972214 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:41.972106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:42.040565 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:42.040534    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:42.141357 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:42.141321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:42.664656 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:42.664629    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cqqkh_aaa9a50a-44c0-4d90-9a3b-1855ab51b369/serve-healthcheck-canary/0.log"
Sep 15 13:39:43.317541 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:43.317510    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:43.384759 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:43.384726    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:43.455255 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:43.455221    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:43.556714 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:43.556679    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:43.915552 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:43.915513    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:44.016326 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:44.016296    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:44.545190 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:44.545158    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:44.944541 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:44.944510    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zp7l2"
Sep 15 13:39:45.625287 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:45.625245    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:45.680455 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:45.680416    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:45.680633 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:45.680472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:45.680633 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:45.680571    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 13:39:45.680633 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:45.680633    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls podName:163705e9-255b-49b1-93bf-e0a56c9fa48e nodeName:}" failed. No retries permitted until 2026-09-15 13:39:53.680617854 +0000 UTC m=+91.558520650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8mlb" (UID: "163705e9-255b-49b1-93bf-e0a56c9fa48e") : secret "insights-runtime-extractor-tls" not found
Sep 15 13:39:45.680834 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:45.680571    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 13:39:45.680834 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:45.680727    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm podName:163705e9-255b-49b1-93bf-e0a56c9fa48e nodeName:}" failed. No retries permitted until 2026-09-15 13:39:53.68070561 +0000 UTC m=+91.558608409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8mlb" (UID: "163705e9-255b-49b1-93bf-e0a56c9fa48e") : configmap "kube-rbac-proxy" not found
Sep 15 13:39:45.721335 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:45.721293    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:45.974769 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:45.974691    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:46.181664 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:46.181630    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:46.637220 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:46.637183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:46.777455 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:46.777423    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:46.915438 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:46.915339    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:46.987508 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:46.987471    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s6mkh"]
Sep 15 13:39:46.990995 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:46.990969    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:46.993201 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:46.993173    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 13:39:46.998161 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:46.994175    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 13:39:46.998161 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:46.994380    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 13:39:46.998161 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:46.994927    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cz7pd\""
Sep 15 13:39:46.998161 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:46.995171    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 13:39:46.998161 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:46.996059    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 13:39:46.998161 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:46.996197    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 13:39:47.017865 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.017841    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:47.091152 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.091116    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16a221f0-a870-4a39-8c0c-db1f8c341ea2-sys\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.091330 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.091190    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16a221f0-a870-4a39-8c0c-db1f8c341ea2-metrics-client-ca\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.091330 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.091216    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76bhg\" (UniqueName: \"kubernetes.io/projected/16a221f0-a870-4a39-8c0c-db1f8c341ea2-kube-api-access-76bhg\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.091330 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.091240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.091483 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.091354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-tls\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.091483 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.091392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.091483 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.091424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16a221f0-a870-4a39-8c0c-db1f8c341ea2-root\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.091483 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.091447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-textfile\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.091679 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.091534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-wtmp\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.192478 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.192396    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16a221f0-a870-4a39-8c0c-db1f8c341ea2-metrics-client-ca\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.192478 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.192431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bhg\" (UniqueName: \"kubernetes.io/projected/16a221f0-a870-4a39-8c0c-db1f8c341ea2-kube-api-access-76bhg\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.192478 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.192453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.192744 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.192503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-tls\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.192744 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.192527    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.192744 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.192553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16a221f0-a870-4a39-8c0c-db1f8c341ea2-root\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.192744 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.192574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-textfile\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.192744 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.192630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-wtmp\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.192744 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.192653    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16a221f0-a870-4a39-8c0c-db1f8c341ea2-sys\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.192744 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:47.192654    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 15 13:39:47.192744 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:39:47.192732    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-tls podName:16a221f0-a870-4a39-8c0c-db1f8c341ea2 nodeName:}" failed. No retries permitted until 2026-09-15 13:39:47.692710295 +0000 UTC m=+85.570613110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-tls") pod "node-exporter-s6mkh" (UID: "16a221f0-a870-4a39-8c0c-db1f8c341ea2") : secret "node-exporter-tls" not found
Sep 15 13:39:47.192744 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.192738    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16a221f0-a870-4a39-8c0c-db1f8c341ea2-sys\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.193190 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.193051    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16a221f0-a870-4a39-8c0c-db1f8c341ea2-metrics-client-ca\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.193245 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.193131    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16a221f0-a870-4a39-8c0c-db1f8c341ea2-root\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.193245 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.193212    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-textfile\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.193245 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.193215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-wtmp\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.193476 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.193458    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.194975 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.194952    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.201309 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.201286    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76bhg\" (UniqueName: \"kubernetes.io/projected/16a221f0-a870-4a39-8c0c-db1f8c341ea2-kube-api-access-76bhg\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.282997 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.282957    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:47.405622 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.405573    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:47.507179 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.507090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:47.612299 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.612268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:47.695293 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.695261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-tls\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.697575 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.697558    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16a221f0-a870-4a39-8c0c-db1f8c341ea2-node-exporter-tls\") pod \"node-exporter-s6mkh\" (UID: \"16a221f0-a870-4a39-8c0c-db1f8c341ea2\") " pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.907633 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.907597    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s6mkh"
Sep 15 13:39:47.919222 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:39:47.919185    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a221f0_a870_4a39_8c0c_db1f8c341ea2.slice/crio-c84ce0d4e6c6e9166b4bdd1acdcaabd7409f374766db85c2d512017fa4752728 WatchSource:0}: Error finding container c84ce0d4e6c6e9166b4bdd1acdcaabd7409f374766db85c2d512017fa4752728: Status 404 returned error can't find the container with id c84ce0d4e6c6e9166b4bdd1acdcaabd7409f374766db85c2d512017fa4752728
Sep 15 13:39:47.954030 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.953994    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:47.994342 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:47.994309    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6mkh" event={"ID":"16a221f0-a870-4a39-8c0c-db1f8c341ea2","Type":"ContainerStarted","Data":"c84ce0d4e6c6e9166b4bdd1acdcaabd7409f374766db85c2d512017fa4752728"}
Sep 15 13:39:48.039331 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:48.039300    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:48.314205 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:48.314110    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:48.807215 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:48.807178    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:48.907364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:48.907328    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:50.075091 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:50.075051    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:50.148909 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:50.148877    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:50.917491 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:50.917461    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:39:51.006956 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:51.006927    2582 generic.go:356] "Generic (PLEG): container finished" podID="16a221f0-a870-4a39-8c0c-db1f8c341ea2" containerID="62c059825f58ff10e9a2a3cca6c722ef63f90713d63c98882162f3cc5c10f2bf" exitCode=0
Sep 15 13:39:51.007117 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:51.006993    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6mkh" event={"ID":"16a221f0-a870-4a39-8c0c-db1f8c341ea2","Type":"ContainerDied","Data":"62c059825f58ff10e9a2a3cca6c722ef63f90713d63c98882162f3cc5c10f2bf"}
Sep 15 13:39:51.249840 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:51.249743    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:51.332680 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:51.332648    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:51.705987 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:51.705953    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:51.798337 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:51.798303    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:52.013540 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:52.013455    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6mkh" event={"ID":"16a221f0-a870-4a39-8c0c-db1f8c341ea2","Type":"ContainerStarted","Data":"d70d585d02cbda005a13d4b7f4aabf90c647571affeeb32f0aebdabf1a604cba"}
Sep 15 13:39:52.013683 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:52.013556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6mkh" event={"ID":"16a221f0-a870-4a39-8c0c-db1f8c341ea2","Type":"ContainerStarted","Data":"30b667adb9f4297b481c039ebc4456418ccdd9c377108d52d4f37186318a59fa"}
Sep 15 13:39:52.115827 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:52.115795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:53.161564 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:53.161534    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:53.747252 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:53.747211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:53.747426 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:53.747287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:53.747895 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:53.747872    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/163705e9-255b-49b1-93bf-e0a56c9fa48e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:53.749577 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:53.749554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/163705e9-255b-49b1-93bf-e0a56c9fa48e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8mlb\" (UID: \"163705e9-255b-49b1-93bf-e0a56c9fa48e\") " pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:53.803715 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:53.803683    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8mlb"
Sep 15 13:39:53.923254 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:53.923220    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:54.022782 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:54.022700    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:54.076794 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:54.076744    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s6mkh" podStartSLOduration=5.514872264 podStartE2EDuration="8.076726402s" totalImagesPullingTime="2.498031961s" totalInitContainerRuntime="63.822177ms" isStatefulPod=true podCreationTimestamp="2026-09-15 13:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:52.041019918 +0000 UTC m=+89.918922738" watchObservedRunningTime="2026-09-15 13:39:54.076726402 +0000 UTC m=+91.954629264"
Sep 15 13:39:54.077253 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:54.077232    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8mlb"]
Sep 15 13:39:54.079970 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:39:54.079944    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163705e9_255b_49b1_93bf_e0a56c9fa48e.slice/crio-b53fa24fdef67cf033a1ba3de5486a8cacd534664d1614c6f7c234959a6830f8 WatchSource:0}: Error finding container b53fa24fdef67cf033a1ba3de5486a8cacd534664d1614c6f7c234959a6830f8: Status 404 returned error can't find the container with id b53fa24fdef67cf033a1ba3de5486a8cacd534664d1614c6f7c234959a6830f8
Sep 15 13:39:54.122932 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:54.122903    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:55.028630 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:55.028592    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8mlb" event={"ID":"163705e9-255b-49b1-93bf-e0a56c9fa48e","Type":"ContainerStarted","Data":"3d43d76d759c2f1ad1e8a1d7bbc043030bdfba826f2003ec62ecc28152396ebe"}
Sep 15 13:39:55.028988 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:55.028677    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8mlb" event={"ID":"163705e9-255b-49b1-93bf-e0a56c9fa48e","Type":"ContainerStarted","Data":"2e4afd9d77e217ce8745b23bba35458b094ea45cfdd6a558a6ec3d972cd7c415"}
Sep 15 13:39:55.028988 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:55.028696    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8mlb" event={"ID":"163705e9-255b-49b1-93bf-e0a56c9fa48e","Type":"ContainerStarted","Data":"b53fa24fdef67cf033a1ba3de5486a8cacd534664d1614c6f7c234959a6830f8"}
Sep 15 13:39:56.170795 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:56.170761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:56.176914 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:56.176882    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57f75d7c55-whpk5"]
Sep 15 13:39:56.276332 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:56.276297    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:56.370919 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:56.370888    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:39:57.038597 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:57.038562    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8mlb" event={"ID":"163705e9-255b-49b1-93bf-e0a56c9fa48e","Type":"ContainerStarted","Data":"fb154578d40f2d4c8c4de03c5f9544025b8442a61ad25c55d0bf53fc155cb434"}
Sep 15 13:39:57.056512 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:57.056460    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l8mlb" podStartSLOduration=17.927113852 podStartE2EDuration="20.056440517s" totalImagesPullingTime="2.129326665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:39:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:39:57.055380219 +0000 UTC m=+94.933283037" watchObservedRunningTime="2026-09-15 13:39:57.056440517 +0000 UTC m=+94.934343336"
Sep 15 13:39:59.906334 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:39:59.906300    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j5pmd"
Sep 15 13:40:00.703872 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:00.703841    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:00.763275 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:00.763243    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:03.108161 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:03.108111    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:03.204333 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:03.204305    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:03.739382 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:03.739341    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:03.797237 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:03.797206    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:05.127212 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:05.127178    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:05.187252 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:05.187223    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:08.933983 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:08.933950    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:09.028387 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:09.028355    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:09.090590 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:09.090554    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:10.084852 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:10.084820    2582 generic.go:356] "Generic (PLEG): container finished" podID="34ab498e-e299-4cab-85a5-d295962eadc8" containerID="91d0599cf484c858bcf8c9aa988471e641bf9b4e1b2fd3ee9498e1c55cb374c8" exitCode=0
Sep 15 13:40:10.085202 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:10.084883    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44" event={"ID":"34ab498e-e299-4cab-85a5-d295962eadc8","Type":"ContainerDied","Data":"91d0599cf484c858bcf8c9aa988471e641bf9b4e1b2fd3ee9498e1c55cb374c8"}
Sep 15 13:40:10.085249 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:10.085233    2582 scope.go:117] "RemoveContainer" containerID="91d0599cf484c858bcf8c9aa988471e641bf9b4e1b2fd3ee9498e1c55cb374c8"
Sep 15 13:40:10.344523 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:10.344497    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:10.403347 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:10.403306    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:11.090705 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:11.090662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rrr44" event={"ID":"34ab498e-e299-4cab-85a5-d295962eadc8","Type":"ContainerStarted","Data":"bdc33153ecac49cea0cc61dea13a84761d4f1985452b3d01fa5ff008f93a014d"}
Sep 15 13:40:17.651272 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:17.651234    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:18.761078 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:18.761045    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:18.861433 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:18.861405    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:21.199096 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.199048    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57f75d7c55-whpk5" podUID="aa7df37b-0f61-4c8e-849c-c680261dc753" containerName="registry" containerID="cri-o://f295f0e38947eefa4efd171e0c3e76d74345747d791c08ebd571cad150a3c951" gracePeriod=30
Sep 15 13:40:21.429725 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.429699    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:40:21.442766 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.442731    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-bound-sa-token\") pod \"aa7df37b-0f61-4c8e-849c-c680261dc753\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") "
Sep 15 13:40:21.442932 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.442811    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa7df37b-0f61-4c8e-849c-c680261dc753-image-registry-private-configuration\") pod \"aa7df37b-0f61-4c8e-849c-c680261dc753\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") "
Sep 15 13:40:21.442932 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.442839    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa7df37b-0f61-4c8e-849c-c680261dc753-ca-trust-extracted\") pod \"aa7df37b-0f61-4c8e-849c-c680261dc753\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") "
Sep 15 13:40:21.442932 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.442864    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls\") pod \"aa7df37b-0f61-4c8e-849c-c680261dc753\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") "
Sep 15 13:40:21.442932 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.442889    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa7df37b-0f61-4c8e-849c-c680261dc753-trusted-ca\") pod \"aa7df37b-0f61-4c8e-849c-c680261dc753\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") "
Sep 15 13:40:21.442932 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.442909    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-certificates\") pod \"aa7df37b-0f61-4c8e-849c-c680261dc753\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") "
Sep 15 13:40:21.443555 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.443225    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz5pq\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-kube-api-access-tz5pq\") pod \"aa7df37b-0f61-4c8e-849c-c680261dc753\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") "
Sep 15 13:40:21.443555 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.443276    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa7df37b-0f61-4c8e-849c-c680261dc753-installation-pull-secrets\") pod \"aa7df37b-0f61-4c8e-849c-c680261dc753\" (UID: \"aa7df37b-0f61-4c8e-849c-c680261dc753\") "
Sep 15 13:40:21.443555 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.443373    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "aa7df37b-0f61-4c8e-849c-c680261dc753" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 13:40:21.443555 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.443383    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7df37b-0f61-4c8e-849c-c680261dc753-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "aa7df37b-0f61-4c8e-849c-c680261dc753" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 13:40:21.443555 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.443510    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa7df37b-0f61-4c8e-849c-c680261dc753-trusted-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:40:21.443555 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.443532    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-certificates\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:40:21.445775 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.445702    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-kube-api-access-tz5pq" (OuterVolumeSpecName: "kube-api-access-tz5pq") pod "aa7df37b-0f61-4c8e-849c-c680261dc753" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753"). InnerVolumeSpecName "kube-api-access-tz5pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:40:21.445899 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.445853    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "aa7df37b-0f61-4c8e-849c-c680261dc753" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:40:21.445899 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.445887    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7df37b-0f61-4c8e-849c-c680261dc753-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "aa7df37b-0f61-4c8e-849c-c680261dc753" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:40:21.446021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.445964    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "aa7df37b-0f61-4c8e-849c-c680261dc753" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:40:21.446021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.445986    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7df37b-0f61-4c8e-849c-c680261dc753-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "aa7df37b-0f61-4c8e-849c-c680261dc753" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:40:21.452349 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.452323    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7df37b-0f61-4c8e-849c-c680261dc753-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "aa7df37b-0f61-4c8e-849c-c680261dc753" (UID: "aa7df37b-0f61-4c8e-849c-c680261dc753"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:40:21.544691 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.544643    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa7df37b-0f61-4c8e-849c-c680261dc753-image-registry-private-configuration\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:40:21.544691 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.544680    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa7df37b-0f61-4c8e-849c-c680261dc753-ca-trust-extracted\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:40:21.544691 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.544690    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-registry-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:40:21.544691 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.544699    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz5pq\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-kube-api-access-tz5pq\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:40:21.544960 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.544708    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa7df37b-0f61-4c8e-849c-c680261dc753-installation-pull-secrets\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:40:21.544960 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:21.544717    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa7df37b-0f61-4c8e-849c-c680261dc753-bound-sa-token\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:40:22.130928 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:22.130890    2582 generic.go:356] "Generic (PLEG): container finished" podID="aa7df37b-0f61-4c8e-849c-c680261dc753" containerID="f295f0e38947eefa4efd171e0c3e76d74345747d791c08ebd571cad150a3c951" exitCode=0
Sep 15 13:40:22.131101 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:22.130951    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57f75d7c55-whpk5"
Sep 15 13:40:22.131101 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:22.130979    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57f75d7c55-whpk5" event={"ID":"aa7df37b-0f61-4c8e-849c-c680261dc753","Type":"ContainerDied","Data":"f295f0e38947eefa4efd171e0c3e76d74345747d791c08ebd571cad150a3c951"}
Sep 15 13:40:22.131101 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:22.131030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57f75d7c55-whpk5" event={"ID":"aa7df37b-0f61-4c8e-849c-c680261dc753","Type":"ContainerDied","Data":"f0f3e596901e56fc0372c32b38d11d5a6a1cdc7e01a4821a60436617ac42de0c"}
Sep 15 13:40:22.131101 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:22.131050    2582 scope.go:117] "RemoveContainer" containerID="f295f0e38947eefa4efd171e0c3e76d74345747d791c08ebd571cad150a3c951"
Sep 15 13:40:22.140530 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:22.140511    2582 scope.go:117] "RemoveContainer" containerID="f295f0e38947eefa4efd171e0c3e76d74345747d791c08ebd571cad150a3c951"
Sep 15 13:40:22.140842 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:40:22.140821    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f295f0e38947eefa4efd171e0c3e76d74345747d791c08ebd571cad150a3c951\": container with ID starting with f295f0e38947eefa4efd171e0c3e76d74345747d791c08ebd571cad150a3c951 not found: ID does not exist" containerID="f295f0e38947eefa4efd171e0c3e76d74345747d791c08ebd571cad150a3c951"
Sep 15 13:40:22.140891 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:22.140852    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f295f0e38947eefa4efd171e0c3e76d74345747d791c08ebd571cad150a3c951"} err="failed to get container status \"f295f0e38947eefa4efd171e0c3e76d74345747d791c08ebd571cad150a3c951\": rpc error: code = NotFound desc = could not find container \"f295f0e38947eefa4efd171e0c3e76d74345747d791c08ebd571cad150a3c951\": container with ID starting with f295f0e38947eefa4efd171e0c3e76d74345747d791c08ebd571cad150a3c951 not found: ID does not exist"
Sep 15 13:40:22.155614 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:22.155585    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57f75d7c55-whpk5"]
Sep 15 13:40:22.159378 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:22.159357    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57f75d7c55-whpk5"]
Sep 15 13:40:22.642082 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:22.642048    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa7df37b-0f61-4c8e-849c-c680261dc753" path="/var/lib/kubelet/pods/aa7df37b-0f61-4c8e-849c-c680261dc753/volumes"
Sep 15 13:40:27.092643 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:27.092608    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:27.185487 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:27.185455    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:27.301116 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:27.301086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:33.711371 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:33.711337    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:33.805676 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:33.805643    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:40:47.245115 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:40:47.245074    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:41:06.060033 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:41:06.059995    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:41:06.617383 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:41:06.617347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:41:08.034996 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:41:08.034965    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:41:08.096417 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:41:08.096381    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:41:15.757470 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:41:15.757390    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:41:15.859603 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:41:15.859575    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:41:15.918727 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:41:15.918696    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:41:16.524601 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:41:16.524570    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:41:16.624536 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:41:16.624506    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:41:26.739583 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:41:26.739548    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:41:26.792805 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:41:26.792775    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:41:26.857477 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:41:26.857445    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:42:21.298054 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:42:21.298019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:42:21.355082 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:42:21.355052    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:42:33.262285 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:42:33.262229    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:42:33.319989 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:42:33.319956    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:42:41.660990 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:42:41.660946    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:42:41.715415 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:42:41.715383    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:42:48.721474 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:42:48.721395    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:42:53.560803 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:42:53.560767    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:17.543023 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:17.542982    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:22.515017 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:22.514992    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 13:43:22.834041 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:22.833856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:22.935218 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:22.935185    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:37.909489 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:37.909426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:38.004948 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:38.004918    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:49.599195 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:49.599122    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:50.464133 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:50.464093    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:53.214972 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:53.214940    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:53.314036 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:53.314001    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:56.088469 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:56.088409    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:56.185580 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:56.185511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:56.583253 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:56.583157    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:56.721636 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:56.721590    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:58.099405 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:58.099364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:58.488845 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:58.488753    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:58.875961 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:58.875928    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9"]
Sep 15 13:43:58.876278 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:58.876259    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7df37b-0f61-4c8e-849c-c680261dc753" containerName="registry"
Sep 15 13:43:58.876369 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:58.876279    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7df37b-0f61-4c8e-849c-c680261dc753" containerName="registry"
Sep 15 13:43:58.876369 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:58.876343    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa7df37b-0f61-4c8e-849c-c680261dc753" containerName="registry"
Sep 15 13:43:58.878234 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:58.878213    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9"
Sep 15 13:43:58.880508 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:58.880488    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 13:43:58.880626 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:58.880518    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sqjlj\""
Sep 15 13:43:58.881461 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:58.881444    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 13:43:58.889930 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:58.889907    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9"]
Sep 15 13:43:58.975529 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:58.975488    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s86fv\" (UniqueName: \"kubernetes.io/projected/2db6d63e-82c4-4ac5-a3bf-8640ef23e621-kube-api-access-s86fv\") pod \"cert-manager-cainjector-559cdd8c87-nt8l9\" (UID: \"2db6d63e-82c4-4ac5-a3bf-8640ef23e621\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9"
Sep 15 13:43:58.975529 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:58.975526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2db6d63e-82c4-4ac5-a3bf-8640ef23e621-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nt8l9\" (UID: \"2db6d63e-82c4-4ac5-a3bf-8640ef23e621\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9"
Sep 15 13:43:59.005880 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:59.005853    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:59.076197 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:59.076167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s86fv\" (UniqueName: \"kubernetes.io/projected/2db6d63e-82c4-4ac5-a3bf-8640ef23e621-kube-api-access-s86fv\") pod \"cert-manager-cainjector-559cdd8c87-nt8l9\" (UID: \"2db6d63e-82c4-4ac5-a3bf-8640ef23e621\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9"
Sep 15 13:43:59.076342 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:59.076208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2db6d63e-82c4-4ac5-a3bf-8640ef23e621-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nt8l9\" (UID: \"2db6d63e-82c4-4ac5-a3bf-8640ef23e621\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9"
Sep 15 13:43:59.084333 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:59.084311    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2db6d63e-82c4-4ac5-a3bf-8640ef23e621-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nt8l9\" (UID: \"2db6d63e-82c4-4ac5-a3bf-8640ef23e621\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9"
Sep 15 13:43:59.084509 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:59.084490    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s86fv\" (UniqueName: \"kubernetes.io/projected/2db6d63e-82c4-4ac5-a3bf-8640ef23e621-kube-api-access-s86fv\") pod \"cert-manager-cainjector-559cdd8c87-nt8l9\" (UID: \"2db6d63e-82c4-4ac5-a3bf-8640ef23e621\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9"
Sep 15 13:43:59.106964 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:59.106933    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:43:59.186462 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:59.186386    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9"
Sep 15 13:43:59.453027 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:59.452841    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9"]
Sep 15 13:43:59.455665 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:43:59.455637    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2db6d63e_82c4_4ac5_a3bf_8640ef23e621.slice/crio-54930d9770929d7fac2fa80962ce75f052ba3c692997e990fc6f73aee22c21ff WatchSource:0}: Error finding container 54930d9770929d7fac2fa80962ce75f052ba3c692997e990fc6f73aee22c21ff: Status 404 returned error can't find the container with id 54930d9770929d7fac2fa80962ce75f052ba3c692997e990fc6f73aee22c21ff
Sep 15 13:43:59.457540 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:59.457518    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 13:43:59.855748 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:43:59.855714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9" event={"ID":"2db6d63e-82c4-4ac5-a3bf-8640ef23e621","Type":"ContainerStarted","Data":"54930d9770929d7fac2fa80962ce75f052ba3c692997e990fc6f73aee22c21ff"}
Sep 15 13:44:05.881340 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:05.881305    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9" event={"ID":"2db6d63e-82c4-4ac5-a3bf-8640ef23e621","Type":"ContainerStarted","Data":"b934bc1a37fd4423e87d6266850b3290e6b591ca0aa37379d829cf63f1c11410"}
Sep 15 13:44:05.897631 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:05.897585    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nt8l9" podStartSLOduration=2.47846667 podStartE2EDuration="7.897569771s" totalImagesPullingTime="5.419103101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:44:05.89649553 +0000 UTC m=+343.774398359" watchObservedRunningTime="2026-09-15 13:44:05.897569771 +0000 UTC m=+343.775472589"
Sep 15 13:44:07.289208 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:07.289158    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:07.498183 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:07.498126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:07.594389 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:07.594359    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:08.085339 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:08.085308    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:08.166454 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:08.166424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:15.038000 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:15.037900    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:15.138274 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:15.138229    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:16.761551 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:16.761504    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:31.861229 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:31.861162    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:31.962187 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:31.962153    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:32.369024 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.368983    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:32.661098 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.661017    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:32.783108 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.783069    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:32.853175 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.853125    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"]
Sep 15 13:44:32.857744 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.857726    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:32.860591 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.860568    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 13:44:32.860760 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.860576    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 13:44:32.860760 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.860680    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p7czt\""
Sep 15 13:44:32.861122 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.861107    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 13:44:32.861122 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.861114    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 13:44:32.878843 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.878820    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"]
Sep 15 13:44:32.910031 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.910000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-webhook-cert\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:32.910179 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.910041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-manifests\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:32.910179 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.910056    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-tmp\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:32.910179 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.910167    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8vmm\" (UniqueName: \"kubernetes.io/projected/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-kube-api-access-f8vmm\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:32.910337 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.910189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:32.978429 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:32.978355    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:33.011155 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.011117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-webhook-cert\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:33.011307 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.011183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-manifests\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:33.011307 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.011201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-tmp\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:33.011307 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.011244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8vmm\" (UniqueName: \"kubernetes.io/projected/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-kube-api-access-f8vmm\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:33.011307 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.011265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:33.011667 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.011646    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-manifests\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:33.013344 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.013322    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-tmp\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:33.013592 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.013572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:33.013655 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.013579    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-webhook-cert\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:33.023507 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.023488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8vmm\" (UniqueName: \"kubernetes.io/projected/b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db-kube-api-access-f8vmm\") pod \"opendatahub-operator-controller-manager-79c64c4b4d-4kwl9\" (UID: \"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db\") " pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:33.167803 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.167753    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:33.199693 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.199655    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:33.300569 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.300491    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:33.364924 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.364894    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:33.461089 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.461066    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"]
Sep 15 13:44:33.463086 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:44:33.463054    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bc75c5_1cbd_48ba_b0ca_aa52eee9e3db.slice/crio-4cee788b96b85483f02f79b0eecb771ed2757d5237f97c05b8dc7642fe44c9aa WatchSource:0}: Error finding container 4cee788b96b85483f02f79b0eecb771ed2757d5237f97c05b8dc7642fe44c9aa: Status 404 returned error can't find the container with id 4cee788b96b85483f02f79b0eecb771ed2757d5237f97c05b8dc7642fe44c9aa
Sep 15 13:44:33.976287 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:33.976247    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9" event={"ID":"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db","Type":"ContainerStarted","Data":"4cee788b96b85483f02f79b0eecb771ed2757d5237f97c05b8dc7642fe44c9aa"}
Sep 15 13:44:36.988791 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:36.988752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9" event={"ID":"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db","Type":"ContainerStarted","Data":"4720117a3c2cef919b446fd34d95a074321f7ac898ac07ccd252f0f1a2a40740"}
Sep 15 13:44:37.993422 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:37.993386    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db" containerID="4720117a3c2cef919b446fd34d95a074321f7ac898ac07ccd252f0f1a2a40740" exitCode=0
Sep 15 13:44:37.993778 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:37.993448    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9" event={"ID":"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db","Type":"ContainerDied","Data":"4720117a3c2cef919b446fd34d95a074321f7ac898ac07ccd252f0f1a2a40740"}
Sep 15 13:44:41.006061 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:41.006026    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9" event={"ID":"b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db","Type":"ContainerStarted","Data":"ecdf4e5407a528e4cccbc1e11a695f1722f8b9e71c0c35b16f1c2ec6ad7fe84b"}
Sep 15 13:44:41.006460 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:41.006125    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:41.029091 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:41.029033    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9" podStartSLOduration=2.754123738 podStartE2EDuration="9.029013121s" totalImagesPullingTime="5.260936112s" totalInitContainerRuntime="1.013953271s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:44:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:44:41.028649202 +0000 UTC m=+378.906552020" watchObservedRunningTime="2026-09-15 13:44:41.029013121 +0000 UTC m=+378.906915941"
Sep 15 13:44:44.666366 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:44.666332    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:44.723416 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:44.723375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:52.011859 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:52.011828    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-79c64c4b4d-4kwl9"
Sep 15 13:44:53.662370 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:53.662328    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:53.793429 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:53.793396    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:54.011689 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.011598    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:54.055758 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.055726    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"]
Sep 15 13:44:54.061678 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.061654    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"
Sep 15 13:44:54.065341 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.065320    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 13:44:54.065487 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.065341    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 13:44:54.065487 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.065345    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-s6979\""
Sep 15 13:44:54.065906 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.065881    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 13:44:54.067545 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.067528    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 13:44:54.085983 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.085954    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"]
Sep 15 13:44:54.174577 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.174548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba8aed7-c12c-4b6f-ae5e-c1445cd935ca-tls-certs\") pod \"kube-auth-proxy-698b9db946-8qhfv\" (UID: \"bba8aed7-c12c-4b6f-ae5e-c1445cd935ca\") " pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"
Sep 15 13:44:54.174748 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.174586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txf8t\" (UniqueName: \"kubernetes.io/projected/bba8aed7-c12c-4b6f-ae5e-c1445cd935ca-kube-api-access-txf8t\") pod \"kube-auth-proxy-698b9db946-8qhfv\" (UID: \"bba8aed7-c12c-4b6f-ae5e-c1445cd935ca\") " pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"
Sep 15 13:44:54.174748 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.174623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bba8aed7-c12c-4b6f-ae5e-c1445cd935ca-tmp\") pod \"kube-auth-proxy-698b9db946-8qhfv\" (UID: \"bba8aed7-c12c-4b6f-ae5e-c1445cd935ca\") " pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"
Sep 15 13:44:54.200476 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.200443    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:54.275331 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.275234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bba8aed7-c12c-4b6f-ae5e-c1445cd935ca-tmp\") pod \"kube-auth-proxy-698b9db946-8qhfv\" (UID: \"bba8aed7-c12c-4b6f-ae5e-c1445cd935ca\") " pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"
Sep 15 13:44:54.275331 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.275297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba8aed7-c12c-4b6f-ae5e-c1445cd935ca-tls-certs\") pod \"kube-auth-proxy-698b9db946-8qhfv\" (UID: \"bba8aed7-c12c-4b6f-ae5e-c1445cd935ca\") " pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"
Sep 15 13:44:54.275516 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.275337    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txf8t\" (UniqueName: \"kubernetes.io/projected/bba8aed7-c12c-4b6f-ae5e-c1445cd935ca-kube-api-access-txf8t\") pod \"kube-auth-proxy-698b9db946-8qhfv\" (UID: \"bba8aed7-c12c-4b6f-ae5e-c1445cd935ca\") " pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"
Sep 15 13:44:54.277514 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.277479    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bba8aed7-c12c-4b6f-ae5e-c1445cd935ca-tmp\") pod \"kube-auth-proxy-698b9db946-8qhfv\" (UID: \"bba8aed7-c12c-4b6f-ae5e-c1445cd935ca\") " pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"
Sep 15 13:44:54.277809 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.277789    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba8aed7-c12c-4b6f-ae5e-c1445cd935ca-tls-certs\") pod \"kube-auth-proxy-698b9db946-8qhfv\" (UID: \"bba8aed7-c12c-4b6f-ae5e-c1445cd935ca\") " pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"
Sep 15 13:44:54.304954 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.304920    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txf8t\" (UniqueName: \"kubernetes.io/projected/bba8aed7-c12c-4b6f-ae5e-c1445cd935ca-kube-api-access-txf8t\") pod \"kube-auth-proxy-698b9db946-8qhfv\" (UID: \"bba8aed7-c12c-4b6f-ae5e-c1445cd935ca\") " pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"
Sep 15 13:44:54.371769 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.371725    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"
Sep 15 13:44:54.645763 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.645745    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-698b9db946-8qhfv"]
Sep 15 13:44:54.648025 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:44:54.647993    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba8aed7_c12c_4b6f_ae5e_c1445cd935ca.slice/crio-3f6e9ef613e09e6868ff7923c126780a73b7d776a46b67cd4178562cfd0cb7c9 WatchSource:0}: Error finding container 3f6e9ef613e09e6868ff7923c126780a73b7d776a46b67cd4178562cfd0cb7c9: Status 404 returned error can't find the container with id 3f6e9ef613e09e6868ff7923c126780a73b7d776a46b67cd4178562cfd0cb7c9
Sep 15 13:44:54.674359 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.674323    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:54.778380 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:54.778343    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:55.055433 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.055347    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv" event={"ID":"bba8aed7-c12c-4b6f-ae5e-c1445cd935ca","Type":"ContainerStarted","Data":"3f6e9ef613e09e6868ff7923c126780a73b7d776a46b67cd4178562cfd0cb7c9"}
Sep 15 13:44:55.552736 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.552685    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:55.566280 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.566223    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"]
Sep 15 13:44:55.569186 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.569159    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.571588 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.571563    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 13:44:55.571989 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.571803    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 15 13:44:55.572106 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.571815    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 15 13:44:55.572106 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.571814    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 15 13:44:55.572106 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.571828    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-l8rhm\""
Sep 15 13:44:55.584636 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.584326    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"]
Sep 15 13:44:55.650234 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.650192    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:55.685589 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.685529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.686039 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.685604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.686039 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.685654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b541f32a-49ce-4e42-865e-a10dc95e21c4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.686039 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.685753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.686039 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.685798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.686039 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.685831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.686039 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.685911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9rbd\" (UniqueName: \"kubernetes.io/projected/b541f32a-49ce-4e42-865e-a10dc95e21c4-kube-api-access-b9rbd\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.787007 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.786969    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.787188 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.787014    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.787188 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.787043    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b541f32a-49ce-4e42-865e-a10dc95e21c4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.787188 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.787095    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.787188 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.787126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.787188 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.787174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.787539 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.787235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9rbd\" (UniqueName: \"kubernetes.io/projected/b541f32a-49ce-4e42-865e-a10dc95e21c4-kube-api-access-b9rbd\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.788058 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.788019    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.790027 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.790002    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.790128 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.790049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b541f32a-49ce-4e42-865e-a10dc95e21c4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.790128 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.790051    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.790282 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.790259    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.803946 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.803890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.812887 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.812862    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9rbd\" (UniqueName: \"kubernetes.io/projected/b541f32a-49ce-4e42-865e-a10dc95e21c4-kube-api-access-b9rbd\") pod \"istiod-openshift-gateway-7b7995d5cd-48zf2\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:55.882880 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:55.882842    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:44:56.518562 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:56.518519    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:44:56.863353 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:56.863316    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"]
Sep 15 13:44:57.318816 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:44:57.318749    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb541f32a_49ce_4e42_865e_a10dc95e21c4.slice/crio-76b91c8c1f1a4934191f170e858d9ed9b928f66b1559be1be0d0240911410562 WatchSource:0}: Error finding container 76b91c8c1f1a4934191f170e858d9ed9b928f66b1559be1be0d0240911410562: Status 404 returned error can't find the container with id 76b91c8c1f1a4934191f170e858d9ed9b928f66b1559be1be0d0240911410562
Sep 15 13:44:58.070210 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:58.070175    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2" event={"ID":"b541f32a-49ce-4e42-865e-a10dc95e21c4","Type":"ContainerStarted","Data":"76b91c8c1f1a4934191f170e858d9ed9b928f66b1559be1be0d0240911410562"}
Sep 15 13:44:58.071635 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:58.071603    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv" event={"ID":"bba8aed7-c12c-4b6f-ae5e-c1445cd935ca","Type":"ContainerStarted","Data":"6e7d5fa2edbccc6788735a42766702e605944c361832cc5306447e0634787c33"}
Sep 15 13:44:58.087481 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:58.087428    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-698b9db946-8qhfv" podStartSLOduration=1.375086297 podStartE2EDuration="4.087412187s" totalImagesPullingTime="2.71232589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:44:58.087019681 +0000 UTC m=+395.964922500" watchObservedRunningTime="2026-09-15 13:44:58.087412187 +0000 UTC m=+395.965315005"
Sep 15 13:44:59.911714 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:59.911671    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:44:59.912093 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:44:59.911756    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:45:00.083061 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:00.083023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2" event={"ID":"b541f32a-49ce-4e42-865e-a10dc95e21c4","Type":"ContainerStarted","Data":"fe351f383e8527cb824ed8fab027fd94c79ab8c03648aff1c9984db326e53823"}
Sep 15 13:45:00.083271 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:00.083193    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:45:00.084778 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:00.084752    2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-48zf2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 13:45:00.084893 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:00.084798    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2" podUID="b541f32a-49ce-4e42-865e-a10dc95e21c4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 13:45:00.102279 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:00.102234    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2" podStartSLOduration=2.511788113 podStartE2EDuration="5.102216951s" totalImagesPullingTime="2.590428838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:45:00.101010952 +0000 UTC m=+397.978913770" watchObservedRunningTime="2026-09-15 13:45:00.102216951 +0000 UTC m=+397.980119770"
Sep 15 13:45:00.721702 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:00.721663    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:00.824855 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:00.824779    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:01.088021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:01.087995    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:45:01.260997 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:01.260957    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:01.431176 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:01.431067    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:01.530542 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:01.530501    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:04.911588 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:04.911555    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:05.015484 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:05.015441    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:05.229587 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:05.229502    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:05.683477 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:05.683447    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:05.783501 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:05.783460    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:05.885525 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:05.885489    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:14.265561 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.265521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:14.361451 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.361408    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t7n99"]
Sep 15 13:45:14.365292 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.365264    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7n99"
Sep 15 13:45:14.367809 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.367784    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 15 13:45:14.370767 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.370745    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-btqzb\""
Sep 15 13:45:14.380912 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.380872    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t7n99"]
Sep 15 13:45:14.381542 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.381518    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq"]
Sep 15 13:45:14.385236 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.385205    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq"
Sep 15 13:45:14.387998 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.387973    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 15 13:45:14.388110 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.387983    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ppxf6\""
Sep 15 13:45:14.395343 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.395316    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:14.399876 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.399851    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq"]
Sep 15 13:45:14.546542 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.546450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hhsl\" (UniqueName: \"kubernetes.io/projected/e8068551-ad23-4b9d-afe8-2e14a577c031-kube-api-access-6hhsl\") pod \"kserve-controller-manager-d6f4b65b8-t7n99\" (UID: \"e8068551-ad23-4b9d-afe8-2e14a577c031\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7n99"
Sep 15 13:45:14.546542 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.546489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04c16753-c34a-4659-81df-a44125a3b449-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-55vgq\" (UID: \"04c16753-c34a-4659-81df-a44125a3b449\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq"
Sep 15 13:45:14.546542 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.546518    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnsgs\" (UniqueName: \"kubernetes.io/projected/04c16753-c34a-4659-81df-a44125a3b449-kube-api-access-rnsgs\") pod \"llmisvc-controller-manager-777c6bd7d5-55vgq\" (UID: \"04c16753-c34a-4659-81df-a44125a3b449\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq"
Sep 15 13:45:14.546756 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.546584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8068551-ad23-4b9d-afe8-2e14a577c031-cert\") pod \"kserve-controller-manager-d6f4b65b8-t7n99\" (UID: \"e8068551-ad23-4b9d-afe8-2e14a577c031\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7n99"
Sep 15 13:45:14.647831 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.647789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hhsl\" (UniqueName: \"kubernetes.io/projected/e8068551-ad23-4b9d-afe8-2e14a577c031-kube-api-access-6hhsl\") pod \"kserve-controller-manager-d6f4b65b8-t7n99\" (UID: \"e8068551-ad23-4b9d-afe8-2e14a577c031\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7n99"
Sep 15 13:45:14.647831 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.647836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04c16753-c34a-4659-81df-a44125a3b449-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-55vgq\" (UID: \"04c16753-c34a-4659-81df-a44125a3b449\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq"
Sep 15 13:45:14.648038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.647866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnsgs\" (UniqueName: \"kubernetes.io/projected/04c16753-c34a-4659-81df-a44125a3b449-kube-api-access-rnsgs\") pod \"llmisvc-controller-manager-777c6bd7d5-55vgq\" (UID: \"04c16753-c34a-4659-81df-a44125a3b449\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq"
Sep 15 13:45:14.648038 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.647894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8068551-ad23-4b9d-afe8-2e14a577c031-cert\") pod \"kserve-controller-manager-d6f4b65b8-t7n99\" (UID: \"e8068551-ad23-4b9d-afe8-2e14a577c031\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7n99"
Sep 15 13:45:14.650371 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.650335    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8068551-ad23-4b9d-afe8-2e14a577c031-cert\") pod \"kserve-controller-manager-d6f4b65b8-t7n99\" (UID: \"e8068551-ad23-4b9d-afe8-2e14a577c031\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7n99"
Sep 15 13:45:14.650552 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.650531    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04c16753-c34a-4659-81df-a44125a3b449-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-55vgq\" (UID: \"04c16753-c34a-4659-81df-a44125a3b449\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq"
Sep 15 13:45:14.658699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.658662    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hhsl\" (UniqueName: \"kubernetes.io/projected/e8068551-ad23-4b9d-afe8-2e14a577c031-kube-api-access-6hhsl\") pod \"kserve-controller-manager-d6f4b65b8-t7n99\" (UID: \"e8068551-ad23-4b9d-afe8-2e14a577c031\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7n99"
Sep 15 13:45:14.660732 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.660711    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnsgs\" (UniqueName: \"kubernetes.io/projected/04c16753-c34a-4659-81df-a44125a3b449-kube-api-access-rnsgs\") pod \"llmisvc-controller-manager-777c6bd7d5-55vgq\" (UID: \"04c16753-c34a-4659-81df-a44125a3b449\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq"
Sep 15 13:45:14.675720 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.675687    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7n99"
Sep 15 13:45:14.699866 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.699837    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq"
Sep 15 13:45:14.940271 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.940236    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:14.950585 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.950558    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t7n99"]
Sep 15 13:45:14.952741 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:45:14.952720    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8068551_ad23_4b9d_afe8_2e14a577c031.slice/crio-8064c27a5d9e6b0fdc49b41aaddbf0e331a21d312ddab7c8958d7a6cd428e433 WatchSource:0}: Error finding container 8064c27a5d9e6b0fdc49b41aaddbf0e331a21d312ddab7c8958d7a6cd428e433: Status 404 returned error can't find the container with id 8064c27a5d9e6b0fdc49b41aaddbf0e331a21d312ddab7c8958d7a6cd428e433
Sep 15 13:45:14.981589 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:14.981559    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq"]
Sep 15 13:45:14.983818 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:45:14.983790    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod04c16753_c34a_4659_81df_a44125a3b449.slice/crio-ba9cad18a2605c4b39bae22593dee8d649dd750604fff958a925b063975f0e50 WatchSource:0}: Error finding container ba9cad18a2605c4b39bae22593dee8d649dd750604fff958a925b063975f0e50: Status 404 returned error can't find the container with id ba9cad18a2605c4b39bae22593dee8d649dd750604fff958a925b063975f0e50
Sep 15 13:45:15.075173 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:15.075108    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:15.140658 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:15.140624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq" event={"ID":"04c16753-c34a-4659-81df-a44125a3b449","Type":"ContainerStarted","Data":"ba9cad18a2605c4b39bae22593dee8d649dd750604fff958a925b063975f0e50"}
Sep 15 13:45:15.141847 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:15.141816    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7n99" event={"ID":"e8068551-ad23-4b9d-afe8-2e14a577c031","Type":"ContainerStarted","Data":"8064c27a5d9e6b0fdc49b41aaddbf0e331a21d312ddab7c8958d7a6cd428e433"}
Sep 15 13:45:15.980004 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:15.979970    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:16.036246 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:16.036182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:45:18.155120 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:18.155089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7n99" event={"ID":"e8068551-ad23-4b9d-afe8-2e14a577c031","Type":"ContainerStarted","Data":"eab91273562fe114a9b479ab95f5c4ac95ef535760a336d5b3bc4e1baed540d5"}
Sep 15 13:45:18.155506 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:18.155191    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7n99"
Sep 15 13:45:18.188983 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:18.188916    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7n99" podStartSLOduration=1.861973153 podStartE2EDuration="4.188899084s" totalImagesPullingTime="2.326925931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:45:18.188012454 +0000 UTC m=+416.065915272" watchObservedRunningTime="2026-09-15 13:45:18.188899084 +0000 UTC m=+416.066801903"
Sep 15 13:45:19.137904 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.137868    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"]
Sep 15 13:45:19.138188 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.138153    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2" podUID="b541f32a-49ce-4e42-865e-a10dc95e21c4" containerName="discovery" containerID="cri-o://fe351f383e8527cb824ed8fab027fd94c79ab8c03648aff1c9984db326e53823" gracePeriod=30
Sep 15 13:45:19.389890 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.389826    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:45:19.491026 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.490986    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-kubeconfig\") pod \"b541f32a-49ce-4e42-865e-a10dc95e21c4\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") "
Sep 15 13:45:19.491026 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.491028    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9rbd\" (UniqueName: \"kubernetes.io/projected/b541f32a-49ce-4e42-865e-a10dc95e21c4-kube-api-access-b9rbd\") pod \"b541f32a-49ce-4e42-865e-a10dc95e21c4\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") "
Sep 15 13:45:19.491299 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.491062    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-csr-dns-cert\") pod \"b541f32a-49ce-4e42-865e-a10dc95e21c4\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") "
Sep 15 13:45:19.491299 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.491087    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-token\") pod \"b541f32a-49ce-4e42-865e-a10dc95e21c4\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") "
Sep 15 13:45:19.491299 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.491123    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-cacerts\") pod \"b541f32a-49ce-4e42-865e-a10dc95e21c4\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") "
Sep 15 13:45:19.491299 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.491173    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b541f32a-49ce-4e42-865e-a10dc95e21c4-local-certs\") pod \"b541f32a-49ce-4e42-865e-a10dc95e21c4\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") "
Sep 15 13:45:19.491299 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.491216    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-csr-ca-configmap\") pod \"b541f32a-49ce-4e42-865e-a10dc95e21c4\" (UID: \"b541f32a-49ce-4e42-865e-a10dc95e21c4\") "
Sep 15 13:45:19.491856 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.491820    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b541f32a-49ce-4e42-865e-a10dc95e21c4" (UID: "b541f32a-49ce-4e42-865e-a10dc95e21c4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 13:45:19.493537 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.493480    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b541f32a-49ce-4e42-865e-a10dc95e21c4" (UID: "b541f32a-49ce-4e42-865e-a10dc95e21c4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:45:19.493644 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.493592    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "b541f32a-49ce-4e42-865e-a10dc95e21c4" (UID: "b541f32a-49ce-4e42-865e-a10dc95e21c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:45:19.493927 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.493896    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b541f32a-49ce-4e42-865e-a10dc95e21c4-local-certs" (OuterVolumeSpecName: "local-certs") pod "b541f32a-49ce-4e42-865e-a10dc95e21c4" (UID: "b541f32a-49ce-4e42-865e-a10dc95e21c4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:45:19.494265 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.494233    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-cacerts" (OuterVolumeSpecName: "cacerts") pod "b541f32a-49ce-4e42-865e-a10dc95e21c4" (UID: "b541f32a-49ce-4e42-865e-a10dc95e21c4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:45:19.494359 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.494245    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b541f32a-49ce-4e42-865e-a10dc95e21c4" (UID: "b541f32a-49ce-4e42-865e-a10dc95e21c4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:45:19.494359 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.494338    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b541f32a-49ce-4e42-865e-a10dc95e21c4-kube-api-access-b9rbd" (OuterVolumeSpecName: "kube-api-access-b9rbd") pod "b541f32a-49ce-4e42-865e-a10dc95e21c4" (UID: "b541f32a-49ce-4e42-865e-a10dc95e21c4"). InnerVolumeSpecName "kube-api-access-b9rbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:45:19.592459 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.592424    2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-kubeconfig\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:45:19.592459 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.592455    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9rbd\" (UniqueName: \"kubernetes.io/projected/b541f32a-49ce-4e42-865e-a10dc95e21c4-kube-api-access-b9rbd\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:45:19.592459 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.592464    2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-csr-dns-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:45:19.592677 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.592473    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-token\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:45:19.592677 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.592482    2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b541f32a-49ce-4e42-865e-a10dc95e21c4-cacerts\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:45:19.592677 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.592490    2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b541f32a-49ce-4e42-865e-a10dc95e21c4-local-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:45:19.592677 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:19.592497    2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b541f32a-49ce-4e42-865e-a10dc95e21c4-istio-csr-ca-configmap\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:45:20.166799 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:20.166770    2582 generic.go:356] "Generic (PLEG): container finished" podID="b541f32a-49ce-4e42-865e-a10dc95e21c4" containerID="fe351f383e8527cb824ed8fab027fd94c79ab8c03648aff1c9984db326e53823" exitCode=0
Sep 15 13:45:20.166909 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:20.166819    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2" event={"ID":"b541f32a-49ce-4e42-865e-a10dc95e21c4","Type":"ContainerDied","Data":"fe351f383e8527cb824ed8fab027fd94c79ab8c03648aff1c9984db326e53823"}
Sep 15 13:45:20.166909 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:20.166827    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"
Sep 15 13:45:20.166909 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:20.166867    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2" event={"ID":"b541f32a-49ce-4e42-865e-a10dc95e21c4","Type":"ContainerDied","Data":"76b91c8c1f1a4934191f170e858d9ed9b928f66b1559be1be0d0240911410562"}
Sep 15 13:45:20.166909 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:20.166886    2582 scope.go:117] "RemoveContainer" containerID="fe351f383e8527cb824ed8fab027fd94c79ab8c03648aff1c9984db326e53823"
Sep 15 13:45:20.179389 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:20.179363    2582 scope.go:117] "RemoveContainer" containerID="fe351f383e8527cb824ed8fab027fd94c79ab8c03648aff1c9984db326e53823"
Sep 15 13:45:20.179685 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:45:20.179665    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe351f383e8527cb824ed8fab027fd94c79ab8c03648aff1c9984db326e53823\": container with ID starting with fe351f383e8527cb824ed8fab027fd94c79ab8c03648aff1c9984db326e53823 not found: ID does not exist" containerID="fe351f383e8527cb824ed8fab027fd94c79ab8c03648aff1c9984db326e53823"
Sep 15 13:45:20.179747 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:20.179694    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe351f383e8527cb824ed8fab027fd94c79ab8c03648aff1c9984db326e53823"} err="failed to get container status \"fe351f383e8527cb824ed8fab027fd94c79ab8c03648aff1c9984db326e53823\": rpc error: code = NotFound desc = could not find container \"fe351f383e8527cb824ed8fab027fd94c79ab8c03648aff1c9984db326e53823\": container with ID starting with fe351f383e8527cb824ed8fab027fd94c79ab8c03648aff1c9984db326e53823 not found: ID does not exist"
Sep 15 13:45:20.203822 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:20.203794    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"]
Sep 15 13:45:20.208928 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:20.208906    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48zf2"]
Sep 15 13:45:20.642233 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:20.642201    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b541f32a-49ce-4e42-865e-a10dc95e21c4" path="/var/lib/kubelet/pods/b541f32a-49ce-4e42-865e-a10dc95e21c4/volumes"
Sep 15 13:45:21.172328 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:21.172292    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq" event={"ID":"04c16753-c34a-4659-81df-a44125a3b449","Type":"ContainerStarted","Data":"0f43323539fee7cacf01f63cfc1e0b63180d5ad1400464176dd119db6bc78633"}
Sep 15 13:45:21.172506 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:21.172399    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq"
Sep 15 13:45:21.188273 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:21.188224    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq" podStartSLOduration=2.019080284 podStartE2EDuration="7.188207507s" totalImagesPullingTime="5.169127223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:45:21.187161856 +0000 UTC m=+419.065064667" watchObservedRunningTime="2026-09-15 13:45:21.188207507 +0000 UTC m=+419.066110325"
Sep 15 13:45:49.166203 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:49.166114    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t7n99"
Sep 15 13:45:52.180610 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:52.180578    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-55vgq"
Sep 15 13:45:59.076482 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:45:59.076447    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:05.942561 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:05.942528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:06.039172 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:06.039126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:10.067882 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:10.067849    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:27.118097 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:27.118051    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:29.177963 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.177933    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"]
Sep 15 13:46:29.178352 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.178246    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b541f32a-49ce-4e42-865e-a10dc95e21c4" containerName="discovery"
Sep 15 13:46:29.178352 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.178261    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b541f32a-49ce-4e42-865e-a10dc95e21c4" containerName="discovery"
Sep 15 13:46:29.178352 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.178311    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b541f32a-49ce-4e42-865e-a10dc95e21c4" containerName="discovery"
Sep 15 13:46:29.181018 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.181002    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:29.186172 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.186129    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-r8x9r\""
Sep 15 13:46:29.186401 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.186386    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 13:46:29.186839 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.186821    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 13:46:29.198525 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.198501    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"]
Sep 15 13:46:29.276084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.276046    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7nn\" (UniqueName: \"kubernetes.io/projected/7e9be0ca-0577-400e-948a-8342573e7aa5-kube-api-access-8t7nn\") pod \"limitador-operator-controller-manager-74495c69f8-75lcg\" (UID: \"7e9be0ca-0577-400e-948a-8342573e7aa5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:29.376894 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.376860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7nn\" (UniqueName: \"kubernetes.io/projected/7e9be0ca-0577-400e-948a-8342573e7aa5-kube-api-access-8t7nn\") pod \"limitador-operator-controller-manager-74495c69f8-75lcg\" (UID: \"7e9be0ca-0577-400e-948a-8342573e7aa5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:29.386784 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.386759    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7nn\" (UniqueName: \"kubernetes.io/projected/7e9be0ca-0577-400e-948a-8342573e7aa5-kube-api-access-8t7nn\") pod \"limitador-operator-controller-manager-74495c69f8-75lcg\" (UID: \"7e9be0ca-0577-400e-948a-8342573e7aa5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:29.491355 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.491273    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:29.762785 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.762695    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"]
Sep 15 13:46:29.765878 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:46:29.765852    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9be0ca_0577_400e_948a_8342573e7aa5.slice/crio-5edf686948c3f9688547818ce37a7c97fd905093a268473cbfde6b87001fd896 WatchSource:0}: Error finding container 5edf686948c3f9688547818ce37a7c97fd905093a268473cbfde6b87001fd896: Status 404 returned error can't find the container with id 5edf686948c3f9688547818ce37a7c97fd905093a268473cbfde6b87001fd896
Sep 15 13:46:29.912912 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:29.912877    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:30.445820 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:30.445783    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg" event={"ID":"7e9be0ca-0577-400e-948a-8342573e7aa5","Type":"ContainerStarted","Data":"5edf686948c3f9688547818ce37a7c97fd905093a268473cbfde6b87001fd896"}
Sep 15 13:46:31.176173 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:31.176117    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t295p"]
Sep 15 13:46:31.179985 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:31.179956    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t295p"
Sep 15 13:46:31.183691 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:31.183610    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 15 13:46:31.183838 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:31.183742    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-dn2wd\""
Sep 15 13:46:31.188287 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:31.188255    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t295p"]
Sep 15 13:46:31.291316 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:31.291266    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r98v\" (UniqueName: \"kubernetes.io/projected/6259b4e7-a69b-4912-b3dd-f058fcb58cf6-kube-api-access-5r98v\") pod \"dns-operator-controller-manager-7b84cd7c9c-t295p\" (UID: \"6259b4e7-a69b-4912-b3dd-f058fcb58cf6\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t295p"
Sep 15 13:46:31.392654 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:31.392608    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r98v\" (UniqueName: \"kubernetes.io/projected/6259b4e7-a69b-4912-b3dd-f058fcb58cf6-kube-api-access-5r98v\") pod \"dns-operator-controller-manager-7b84cd7c9c-t295p\" (UID: \"6259b4e7-a69b-4912-b3dd-f058fcb58cf6\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t295p"
Sep 15 13:46:31.411970 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:31.411939    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r98v\" (UniqueName: \"kubernetes.io/projected/6259b4e7-a69b-4912-b3dd-f058fcb58cf6-kube-api-access-5r98v\") pod \"dns-operator-controller-manager-7b84cd7c9c-t295p\" (UID: \"6259b4e7-a69b-4912-b3dd-f058fcb58cf6\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t295p"
Sep 15 13:46:31.450530 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:31.450405    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:31.494036 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:31.493997    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t295p"
Sep 15 13:46:31.711813 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:31.711735    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:31.777370 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:31.777250    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t295p"]
Sep 15 13:46:32.116111 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:32.116071    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:32.355476 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:32.355438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:32.525647 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:32.525563    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:32.574011 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:46:32.573972    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6259b4e7_a69b_4912_b3dd_f058fcb58cf6.slice/crio-182e0b2267133832eff0d8456b2d22c40294a1e7d227037fe674c94b0df6668b WatchSource:0}: Error finding container 182e0b2267133832eff0d8456b2d22c40294a1e7d227037fe674c94b0df6668b: Status 404 returned error can't find the container with id 182e0b2267133832eff0d8456b2d22c40294a1e7d227037fe674c94b0df6668b
Sep 15 13:46:33.462754 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:33.462720    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg" event={"ID":"7e9be0ca-0577-400e-948a-8342573e7aa5","Type":"ContainerStarted","Data":"0082c307b7abf24559b14f1686553e18f4c677820209ef88a1b895eaaf16d202"}
Sep 15 13:46:33.462953 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:33.462798    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:33.464021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:33.463999    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t295p" event={"ID":"6259b4e7-a69b-4912-b3dd-f058fcb58cf6","Type":"ContainerStarted","Data":"182e0b2267133832eff0d8456b2d22c40294a1e7d227037fe674c94b0df6668b"}
Sep 15 13:46:33.485605 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:33.485552    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg" podStartSLOduration=1.611621083 podStartE2EDuration="4.485535188s" totalImagesPullingTime="2.873914105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:46:33.485110493 +0000 UTC m=+491.363013312" watchObservedRunningTime="2026-09-15 13:46:33.485535188 +0000 UTC m=+491.363438008"
Sep 15 13:46:33.569033 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:33.569002    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:34.162278 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:34.162248    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:34.259098 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:34.259065    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:34.591679 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:34.591635    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:34.692177 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:34.692123    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:35.474302 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:35.474268    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t295p" event={"ID":"6259b4e7-a69b-4912-b3dd-f058fcb58cf6","Type":"ContainerStarted","Data":"03b02af731c2692c324fd2934f37c5e8e37ae9591c08bf8fc17c372e08f97e19"}
Sep 15 13:46:35.474473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:35.474339    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t295p"
Sep 15 13:46:35.504364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:35.504311    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t295p" podStartSLOduration=2.218730814 podStartE2EDuration="4.504294214s" totalImagesPullingTime="2.2855634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:46:35.503991172 +0000 UTC m=+493.381893992" watchObservedRunningTime="2026-09-15 13:46:35.504294214 +0000 UTC m=+493.382197032"
Sep 15 13:46:36.061276 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.061243    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"]
Sep 15 13:46:36.064610 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.064592    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:36.067441 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.067422    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m57h2\""
Sep 15 13:46:36.082251 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.082226    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"]
Sep 15 13:46:36.130886 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.130849    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxnht\" (UniqueName: \"kubernetes.io/projected/2df68655-9441-4325-a08c-7c5283226c85-kube-api-access-cxnht\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" (UID: \"2df68655-9441-4325-a08c-7c5283226c85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:36.130886 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.130892    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2df68655-9441-4325-a08c-7c5283226c85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" (UID: \"2df68655-9441-4325-a08c-7c5283226c85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:36.134462 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.134430    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:36.231781 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.231745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxnht\" (UniqueName: \"kubernetes.io/projected/2df68655-9441-4325-a08c-7c5283226c85-kube-api-access-cxnht\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" (UID: \"2df68655-9441-4325-a08c-7c5283226c85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:36.231942 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.231789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2df68655-9441-4325-a08c-7c5283226c85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" (UID: \"2df68655-9441-4325-a08c-7c5283226c85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:36.232122 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.232106    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2df68655-9441-4325-a08c-7c5283226c85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" (UID: \"2df68655-9441-4325-a08c-7c5283226c85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:36.240845 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.240814    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxnht\" (UniqueName: \"kubernetes.io/projected/2df68655-9441-4325-a08c-7c5283226c85-kube-api-access-cxnht\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" (UID: \"2df68655-9441-4325-a08c-7c5283226c85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:36.375082 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.375046    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:36.508840 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.508807    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:36.656123 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:36.656035    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"]
Sep 15 13:46:36.659813 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:46:36.659786    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df68655_9441_4325_a08c_7c5283226c85.slice/crio-6c8ec27980e1a744a210962c01c037fe19a14e79983f876f2cddf6c0af1930d5 WatchSource:0}: Error finding container 6c8ec27980e1a744a210962c01c037fe19a14e79983f876f2cddf6c0af1930d5: Status 404 returned error can't find the container with id 6c8ec27980e1a744a210962c01c037fe19a14e79983f876f2cddf6c0af1930d5
Sep 15 13:46:37.483756 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:37.483723    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t" event={"ID":"2df68655-9441-4325-a08c-7c5283226c85","Type":"ContainerStarted","Data":"6c8ec27980e1a744a210962c01c037fe19a14e79983f876f2cddf6c0af1930d5"}
Sep 15 13:46:38.415892 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:38.415865    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:38.480437 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:38.480403    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-xmn2l"]
Sep 15 13:46:38.483436 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:38.483420    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-xmn2l"
Sep 15 13:46:38.486497 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:38.486474    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jlk7b\""
Sep 15 13:46:38.496547 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:38.496523    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-xmn2l"]
Sep 15 13:46:38.534858 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:38.534824    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:38.551911 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:38.551879    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-975qp\" (UniqueName: \"kubernetes.io/projected/6b412a25-f93d-4c2a-a85e-98487b34ecf0-kube-api-access-975qp\") pod \"authorino-operator-76b478bd78-xmn2l\" (UID: \"6b412a25-f93d-4c2a-a85e-98487b34ecf0\") " pod="kuadrant-system/authorino-operator-76b478bd78-xmn2l"
Sep 15 13:46:38.652719 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:38.652689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-975qp\" (UniqueName: \"kubernetes.io/projected/6b412a25-f93d-4c2a-a85e-98487b34ecf0-kube-api-access-975qp\") pod \"authorino-operator-76b478bd78-xmn2l\" (UID: \"6b412a25-f93d-4c2a-a85e-98487b34ecf0\") " pod="kuadrant-system/authorino-operator-76b478bd78-xmn2l"
Sep 15 13:46:38.666786 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:38.666718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-975qp\" (UniqueName: \"kubernetes.io/projected/6b412a25-f93d-4c2a-a85e-98487b34ecf0-kube-api-access-975qp\") pod \"authorino-operator-76b478bd78-xmn2l\" (UID: \"6b412a25-f93d-4c2a-a85e-98487b34ecf0\") " pod="kuadrant-system/authorino-operator-76b478bd78-xmn2l"
Sep 15 13:46:38.794391 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:38.794352    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-xmn2l"
Sep 15 13:46:39.080906 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:39.080836    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-xmn2l"]
Sep 15 13:46:39.084400 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:46:39.084368    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b412a25_f93d_4c2a_a85e_98487b34ecf0.slice/crio-7220a9475803729d9c2ee2718d87cac276600dc7aa74375d2d34f672c45edb70 WatchSource:0}: Error finding container 7220a9475803729d9c2ee2718d87cac276600dc7aa74375d2d34f672c45edb70: Status 404 returned error can't find the container with id 7220a9475803729d9c2ee2718d87cac276600dc7aa74375d2d34f672c45edb70
Sep 15 13:46:39.501245 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:39.501211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-xmn2l" event={"ID":"6b412a25-f93d-4c2a-a85e-98487b34ecf0","Type":"ContainerStarted","Data":"7220a9475803729d9c2ee2718d87cac276600dc7aa74375d2d34f672c45edb70"}
Sep 15 13:46:39.943247 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:39.943209    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:40.043888 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:40.043860    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:41.513094 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:41.513024    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t" event={"ID":"2df68655-9441-4325-a08c-7c5283226c85","Type":"ContainerStarted","Data":"b5ca5f6e18463950d103332dad382d9a5e8185efe5acf3c4f72881fe98bd18bb"}
Sep 15 13:46:41.513588 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:41.513188    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:41.534304 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:41.534250    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t" podStartSLOduration=0.849336594 podStartE2EDuration="5.534232665s" totalImagesPullingTime="4.684896071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:46:41.532951753 +0000 UTC m=+499.410854564" watchObservedRunningTime="2026-09-15 13:46:41.534232665 +0000 UTC m=+499.412135542"
Sep 15 13:46:41.922782 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:41.922735    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:42.023465 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:42.023433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:42.519024 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:42.518988    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-xmn2l" event={"ID":"6b412a25-f93d-4c2a-a85e-98487b34ecf0","Type":"ContainerStarted","Data":"eb51538dd9c16ba8c5df22ae66acec7a49964b01976a3ada67289920bdc3f15a"}
Sep 15 13:46:42.519407 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:42.519309    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-xmn2l"
Sep 15 13:46:42.537856 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:42.537804    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-xmn2l" podStartSLOduration=1.590421358 podStartE2EDuration="4.537788406s" totalImagesPullingTime="2.947367048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:46:42.537082206 +0000 UTC m=+500.414985037" watchObservedRunningTime="2026-09-15 13:46:42.537788406 +0000 UTC m=+500.415691226"
Sep 15 13:46:44.131733 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.131698    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"]
Sep 15 13:46:44.135102 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.135075    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"
Sep 15 13:46:44.135650 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.135620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:44.137543 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.137485    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 15 13:46:44.137733 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.137492    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 15 13:46:44.137855 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.137838    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7kjhd\""
Sep 15 13:46:44.143414 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.143391    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"]
Sep 15 13:46:44.202504 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.202471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh2mr\" (UniqueName: \"kubernetes.io/projected/58376e94-40ae-4172-ab79-58b90943d587-kube-api-access-vh2mr\") pod \"kuadrant-console-plugin-fbdd4f445-8lpwx\" (UID: \"58376e94-40ae-4172-ab79-58b90943d587\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"
Sep 15 13:46:44.202718 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.202524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/58376e94-40ae-4172-ab79-58b90943d587-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8lpwx\" (UID: \"58376e94-40ae-4172-ab79-58b90943d587\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"
Sep 15 13:46:44.202718 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.202590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/58376e94-40ae-4172-ab79-58b90943d587-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8lpwx\" (UID: \"58376e94-40ae-4172-ab79-58b90943d587\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"
Sep 15 13:46:44.304008 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.303973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/58376e94-40ae-4172-ab79-58b90943d587-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8lpwx\" (UID: \"58376e94-40ae-4172-ab79-58b90943d587\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"
Sep 15 13:46:44.304008 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.304015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/58376e94-40ae-4172-ab79-58b90943d587-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8lpwx\" (UID: \"58376e94-40ae-4172-ab79-58b90943d587\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"
Sep 15 13:46:44.304293 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.304059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh2mr\" (UniqueName: \"kubernetes.io/projected/58376e94-40ae-4172-ab79-58b90943d587-kube-api-access-vh2mr\") pod \"kuadrant-console-plugin-fbdd4f445-8lpwx\" (UID: \"58376e94-40ae-4172-ab79-58b90943d587\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"
Sep 15 13:46:44.304293 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:44.304211    2582 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 15 13:46:44.304293 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:44.304279    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58376e94-40ae-4172-ab79-58b90943d587-plugin-serving-cert podName:58376e94-40ae-4172-ab79-58b90943d587 nodeName:}" failed. No retries permitted until 2026-09-15 13:46:44.804258904 +0000 UTC m=+502.682161719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/58376e94-40ae-4172-ab79-58b90943d587-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-8lpwx" (UID: "58376e94-40ae-4172-ab79-58b90943d587") : secret "plugin-serving-cert" not found
Sep 15 13:46:44.304714 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.304693    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/58376e94-40ae-4172-ab79-58b90943d587-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8lpwx\" (UID: \"58376e94-40ae-4172-ab79-58b90943d587\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"
Sep 15 13:46:44.312623 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.312594    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh2mr\" (UniqueName: \"kubernetes.io/projected/58376e94-40ae-4172-ab79-58b90943d587-kube-api-access-vh2mr\") pod \"kuadrant-console-plugin-fbdd4f445-8lpwx\" (UID: \"58376e94-40ae-4172-ab79-58b90943d587\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"
Sep 15 13:46:44.464605 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.464517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:44.471202 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.471176    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:44.809118 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.809030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/58376e94-40ae-4172-ab79-58b90943d587-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8lpwx\" (UID: \"58376e94-40ae-4172-ab79-58b90943d587\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"
Sep 15 13:46:44.811451 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:44.811417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/58376e94-40ae-4172-ab79-58b90943d587-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8lpwx\" (UID: \"58376e94-40ae-4172-ab79-58b90943d587\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"
Sep 15 13:46:45.045104 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:45.045066    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"
Sep 15 13:46:45.316983 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:45.316961    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx"]
Sep 15 13:46:45.319235 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:46:45.319198    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58376e94_40ae_4172_ab79_58b90943d587.slice/crio-aa4e6c97bb4d21f3fa9f7aa777c0c7db7a3e0ec87ac7f518b87ba616107451fe WatchSource:0}: Error finding container aa4e6c97bb4d21f3fa9f7aa777c0c7db7a3e0ec87ac7f518b87ba616107451fe: Status 404 returned error can't find the container with id aa4e6c97bb4d21f3fa9f7aa777c0c7db7a3e0ec87ac7f518b87ba616107451fe
Sep 15 13:46:45.532201 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:45.532105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx" event={"ID":"58376e94-40ae-4172-ab79-58b90943d587","Type":"ContainerStarted","Data":"aa4e6c97bb4d21f3fa9f7aa777c0c7db7a3e0ec87ac7f518b87ba616107451fe"}
Sep 15 13:46:46.480989 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:46.480955    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t295p"
Sep 15 13:46:48.116443 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:48.116405    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:48.173644 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:48.173611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:48.725406 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:48.725368    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:48.800719 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:48.800676    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:49.430504 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:49.430470    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:49.517330 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:49.517292    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:50.558713 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:50.558633    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx" event={"ID":"58376e94-40ae-4172-ab79-58b90943d587","Type":"ContainerStarted","Data":"3f068f722c3c13f79c5929dcef0dca4f2d52cc40aab16480d142315d1caf589a"}
Sep 15 13:46:50.574218 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:50.574166    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8lpwx" podStartSLOduration=1.740639598 podStartE2EDuration="6.574129932s" totalImagesPullingTime="4.833490334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:46:50.573610793 +0000 UTC m=+508.451513809" watchObservedRunningTime="2026-09-15 13:46:50.574129932 +0000 UTC m=+508.452032752"
Sep 15 13:46:52.521047 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:52.521010    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:53.525565 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:53.525535    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-xmn2l"
Sep 15 13:46:54.382874 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.382835    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"]
Sep 15 13:46:54.387312 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.387292    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:54.405639 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.405608    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"]
Sep 15 13:46:54.488349 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.488293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3624b26-bc20-4a9b-9ca1-320889d71f54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-l5bvc\" (UID: \"c3624b26-bc20-4a9b-9ca1-320889d71f54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:54.488528 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.488380    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnf9s\" (UniqueName: \"kubernetes.io/projected/c3624b26-bc20-4a9b-9ca1-320889d71f54-kube-api-access-jnf9s\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-l5bvc\" (UID: \"c3624b26-bc20-4a9b-9ca1-320889d71f54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:54.589002 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.588967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnf9s\" (UniqueName: \"kubernetes.io/projected/c3624b26-bc20-4a9b-9ca1-320889d71f54-kube-api-access-jnf9s\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-l5bvc\" (UID: \"c3624b26-bc20-4a9b-9ca1-320889d71f54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:54.589394 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.589030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3624b26-bc20-4a9b-9ca1-320889d71f54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-l5bvc\" (UID: \"c3624b26-bc20-4a9b-9ca1-320889d71f54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:54.589502 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.589483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3624b26-bc20-4a9b-9ca1-320889d71f54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-l5bvc\" (UID: \"c3624b26-bc20-4a9b-9ca1-320889d71f54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:54.598321 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.598297    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnf9s\" (UniqueName: \"kubernetes.io/projected/c3624b26-bc20-4a9b-9ca1-320889d71f54-kube-api-access-jnf9s\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-l5bvc\" (UID: \"c3624b26-bc20-4a9b-9ca1-320889d71f54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:54.650881 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.650803    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"]
Sep 15 13:46:54.651017 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.650975    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t" podUID="2df68655-9441-4325-a08c-7c5283226c85" containerName="manager" containerID="cri-o://b5ca5f6e18463950d103332dad382d9a5e8185efe5acf3c4f72881fe98bd18bb" gracePeriod=2
Sep 15 13:46:54.662105 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.662077    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"]
Sep 15 13:46:54.667203 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:54.667016    2582 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2df68655-9441-4325-a08c-7c5283226c85\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\": pods \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:54.691001 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.690971    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"]
Sep 15 13:46:54.691255 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.691241    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:54.702711 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.702689    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"]
Sep 15 13:46:54.703039 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.703026    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df68655-9441-4325-a08c-7c5283226c85" containerName="manager"
Sep 15 13:46:54.703081 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.703041    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df68655-9441-4325-a08c-7c5283226c85" containerName="manager"
Sep 15 13:46:54.703112 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.703095    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df68655-9441-4325-a08c-7c5283226c85" containerName="manager"
Sep 15 13:46:54.706068 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.706050    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"
Sep 15 13:46:54.708747 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:54.708720    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="2df68655-9441-4325-a08c-7c5283226c85" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:54.711578 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.711551    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"]
Sep 15 13:46:54.739249 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.739219    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"]
Sep 15 13:46:54.741126 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.741099    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"]
Sep 15 13:46:54.741362 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.741333    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg" podUID="7e9be0ca-0577-400e-948a-8342573e7aa5" containerName="manager" containerID="cri-o://0082c307b7abf24559b14f1686553e18f4c677820209ef88a1b895eaaf16d202" gracePeriod=2
Sep 15 13:46:54.763263 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.763235    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"]
Sep 15 13:46:54.769460 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.769435    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"]
Sep 15 13:46:54.769864 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.769850    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9be0ca-0577-400e-948a-8342573e7aa5" containerName="manager"
Sep 15 13:46:54.769912 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.769867    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9be0ca-0577-400e-948a-8342573e7aa5" containerName="manager"
Sep 15 13:46:54.769974 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.769962    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9be0ca-0577-400e-948a-8342573e7aa5" containerName="manager"
Sep 15 13:46:54.772834 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.772815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"
Sep 15 13:46:54.790686 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.790651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48zmm\" (UniqueName: \"kubernetes.io/projected/7af63032-a064-448d-865c-7cfdab1cb6c2-kube-api-access-48zmm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-54jdz\" (UID: \"7af63032-a064-448d-865c-7cfdab1cb6c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"
Sep 15 13:46:54.790844 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.790713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7af63032-a064-448d-865c-7cfdab1cb6c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-54jdz\" (UID: \"7af63032-a064-448d-865c-7cfdab1cb6c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"
Sep 15 13:46:54.797557 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.797519    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5djd4"]
Sep 15 13:46:54.800719 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.800697    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5djd4"
Sep 15 13:46:54.817326 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.817299    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"]
Sep 15 13:46:54.828921 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.828894    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5djd4"]
Sep 15 13:46:54.847380 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:54.847343    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-75lcg\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="7e9be0ca-0577-400e-948a-8342573e7aa5" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:54.851286 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:54.851258    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="2df68655-9441-4325-a08c-7c5283226c85" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:54.858059 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:54.858028    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="2df68655-9441-4325-a08c-7c5283226c85" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:54.861627 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:54.861593    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-75lcg\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="7e9be0ca-0577-400e-948a-8342573e7aa5" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:54.892636 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.892032    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc88d72-d3d2-41d7-a03e-009283733208-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-qn5rc\" (UID: \"fdc88d72-d3d2-41d7-a03e-009283733208\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"
Sep 15 13:46:54.892636 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.892089    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqn8b\" (UniqueName: \"kubernetes.io/projected/9e72947c-265d-4fce-a5ad-4a8341bb14da-kube-api-access-dqn8b\") pod \"limitador-operator-controller-manager-74495c69f8-5djd4\" (UID: \"9e72947c-265d-4fce-a5ad-4a8341bb14da\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5djd4"
Sep 15 13:46:54.892636 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.892173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48zmm\" (UniqueName: \"kubernetes.io/projected/7af63032-a064-448d-865c-7cfdab1cb6c2-kube-api-access-48zmm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-54jdz\" (UID: \"7af63032-a064-448d-865c-7cfdab1cb6c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"
Sep 15 13:46:54.892636 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.892218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7af63032-a064-448d-865c-7cfdab1cb6c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-54jdz\" (UID: \"7af63032-a064-448d-865c-7cfdab1cb6c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"
Sep 15 13:46:54.892636 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.892260    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p528m\" (UniqueName: \"kubernetes.io/projected/fdc88d72-d3d2-41d7-a03e-009283733208-kube-api-access-p528m\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-qn5rc\" (UID: \"fdc88d72-d3d2-41d7-a03e-009283733208\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"
Sep 15 13:46:54.893061 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.893032    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7af63032-a064-448d-865c-7cfdab1cb6c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-54jdz\" (UID: \"7af63032-a064-448d-865c-7cfdab1cb6c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"
Sep 15 13:46:54.905100 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.905034    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48zmm\" (UniqueName: \"kubernetes.io/projected/7af63032-a064-448d-865c-7cfdab1cb6c2-kube-api-access-48zmm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-54jdz\" (UID: \"7af63032-a064-448d-865c-7cfdab1cb6c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"
Sep 15 13:46:54.977451 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.977428    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:54.979457 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:54.979434    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-75lcg\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="7e9be0ca-0577-400e-948a-8342573e7aa5" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:54.981089 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:54.981062    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="2df68655-9441-4325-a08c-7c5283226c85" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:54.984280 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.984260    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:54.986377 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:54.986356    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-75lcg\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="7e9be0ca-0577-400e-948a-8342573e7aa5" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:54.988069 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:54.988048    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="2df68655-9441-4325-a08c-7c5283226c85" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:54.993456 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.993436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p528m\" (UniqueName: \"kubernetes.io/projected/fdc88d72-d3d2-41d7-a03e-009283733208-kube-api-access-p528m\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-qn5rc\" (UID: \"fdc88d72-d3d2-41d7-a03e-009283733208\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"
Sep 15 13:46:54.993507 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.993471    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc88d72-d3d2-41d7-a03e-009283733208-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-qn5rc\" (UID: \"fdc88d72-d3d2-41d7-a03e-009283733208\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"
Sep 15 13:46:54.993507 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.993492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqn8b\" (UniqueName: \"kubernetes.io/projected/9e72947c-265d-4fce-a5ad-4a8341bb14da-kube-api-access-dqn8b\") pod \"limitador-operator-controller-manager-74495c69f8-5djd4\" (UID: \"9e72947c-265d-4fce-a5ad-4a8341bb14da\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5djd4"
Sep 15 13:46:54.993769 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:54.993751    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc88d72-d3d2-41d7-a03e-009283733208-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-qn5rc\" (UID: \"fdc88d72-d3d2-41d7-a03e-009283733208\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"
Sep 15 13:46:55.002664 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.002644    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p528m\" (UniqueName: \"kubernetes.io/projected/fdc88d72-d3d2-41d7-a03e-009283733208-kube-api-access-p528m\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-qn5rc\" (UID: \"fdc88d72-d3d2-41d7-a03e-009283733208\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"
Sep 15 13:46:55.002756 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.002678    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqn8b\" (UniqueName: \"kubernetes.io/projected/9e72947c-265d-4fce-a5ad-4a8341bb14da-kube-api-access-dqn8b\") pod \"limitador-operator-controller-manager-74495c69f8-5djd4\" (UID: \"9e72947c-265d-4fce-a5ad-4a8341bb14da\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5djd4"
Sep 15 13:46:55.047812 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.047782    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"
Sep 15 13:46:55.093788 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.093752    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxnht\" (UniqueName: \"kubernetes.io/projected/2df68655-9441-4325-a08c-7c5283226c85-kube-api-access-cxnht\") pod \"2df68655-9441-4325-a08c-7c5283226c85\" (UID: \"2df68655-9441-4325-a08c-7c5283226c85\") "
Sep 15 13:46:55.093961 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.093847    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t7nn\" (UniqueName: \"kubernetes.io/projected/7e9be0ca-0577-400e-948a-8342573e7aa5-kube-api-access-8t7nn\") pod \"7e9be0ca-0577-400e-948a-8342573e7aa5\" (UID: \"7e9be0ca-0577-400e-948a-8342573e7aa5\") "
Sep 15 13:46:55.093961 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.093919    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2df68655-9441-4325-a08c-7c5283226c85-extensions-socket-volume\") pod \"2df68655-9441-4325-a08c-7c5283226c85\" (UID: \"2df68655-9441-4325-a08c-7c5283226c85\") "
Sep 15 13:46:55.094481 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.094449    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df68655-9441-4325-a08c-7c5283226c85-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2df68655-9441-4325-a08c-7c5283226c85" (UID: "2df68655-9441-4325-a08c-7c5283226c85"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:46:55.096767 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.096724    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df68655-9441-4325-a08c-7c5283226c85-kube-api-access-cxnht" (OuterVolumeSpecName: "kube-api-access-cxnht") pod "2df68655-9441-4325-a08c-7c5283226c85" (UID: "2df68655-9441-4325-a08c-7c5283226c85"). InnerVolumeSpecName "kube-api-access-cxnht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:46:55.097396 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.096794    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9be0ca-0577-400e-948a-8342573e7aa5-kube-api-access-8t7nn" (OuterVolumeSpecName: "kube-api-access-8t7nn") pod "7e9be0ca-0577-400e-948a-8342573e7aa5" (UID: "7e9be0ca-0577-400e-948a-8342573e7aa5"). InnerVolumeSpecName "kube-api-access-8t7nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:46:55.129520 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.129487    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"
Sep 15 13:46:55.136185 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.136155    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5djd4"
Sep 15 13:46:55.196462 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.195680    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t7nn\" (UniqueName: \"kubernetes.io/projected/7e9be0ca-0577-400e-948a-8342573e7aa5-kube-api-access-8t7nn\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:46:55.196462 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.195716    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2df68655-9441-4325-a08c-7c5283226c85-extensions-socket-volume\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:46:55.196462 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.195731    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxnht\" (UniqueName: \"kubernetes.io/projected/2df68655-9441-4325-a08c-7c5283226c85-kube-api-access-cxnht\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:46:55.327495 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.327469    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"]
Sep 15 13:46:55.329379 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:46:55.329343    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af63032_a064_448d_865c_7cfdab1cb6c2.slice/crio-aa233353d9ffed9b65572ff736dd2b8bdfdf4342b30774f58ca661bf64a00f9e WatchSource:0}: Error finding container aa233353d9ffed9b65572ff736dd2b8bdfdf4342b30774f58ca661bf64a00f9e: Status 404 returned error can't find the container with id aa233353d9ffed9b65572ff736dd2b8bdfdf4342b30774f58ca661bf64a00f9e
Sep 15 13:46:55.421374 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.421348    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5djd4"]
Sep 15 13:46:55.423285 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:46:55.423258    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e72947c_265d_4fce_a5ad_4a8341bb14da.slice/crio-3ca119c71d0b1473d70a68bd708a671e4bb8edbd15bffbe28ee8edfd9a9ac0f8 WatchSource:0}: Error finding container 3ca119c71d0b1473d70a68bd708a671e4bb8edbd15bffbe28ee8edfd9a9ac0f8: Status 404 returned error can't find the container with id 3ca119c71d0b1473d70a68bd708a671e4bb8edbd15bffbe28ee8edfd9a9ac0f8
Sep 15 13:46:55.427823 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.427797    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"]
Sep 15 13:46:55.431773 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:46:55.431749    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc88d72_d3d2_41d7_a03e_009283733208.slice/crio-60c0c6843d3602b68413319eb1530bf991b7edf8f2543c6217025e37bcbd23de WatchSource:0}: Error finding container 60c0c6843d3602b68413319eb1530bf991b7edf8f2543c6217025e37bcbd23de: Status 404 returned error can't find the container with id 60c0c6843d3602b68413319eb1530bf991b7edf8f2543c6217025e37bcbd23de
Sep 15 13:46:55.584213 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.584177    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc" event={"ID":"fdc88d72-d3d2-41d7-a03e-009283733208","Type":"ContainerStarted","Data":"727ff7d3f3ffb04b4496fb4a1f3d6cd1ddd93110c41898d48c22109c8172fbf6"}
Sep 15 13:46:55.584410 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.584243    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc" event={"ID":"fdc88d72-d3d2-41d7-a03e-009283733208","Type":"ContainerStarted","Data":"60c0c6843d3602b68413319eb1530bf991b7edf8f2543c6217025e37bcbd23de"}
Sep 15 13:46:55.584410 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.584275    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"
Sep 15 13:46:55.586100 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:55.586069    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="2df68655-9441-4325-a08c-7c5283226c85" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:55.586228 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.586102    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz" event={"ID":"7af63032-a064-448d-865c-7cfdab1cb6c2","Type":"ContainerStarted","Data":"4e3a0bd541abc82898f0a48fcc6d99099b0728812140e330a3a565223931cdab"}
Sep 15 13:46:55.586228 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.586188    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"
Sep 15 13:46:55.586228 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.586216    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz" event={"ID":"7af63032-a064-448d-865c-7cfdab1cb6c2","Type":"ContainerStarted","Data":"aa233353d9ffed9b65572ff736dd2b8bdfdf4342b30774f58ca661bf64a00f9e"}
Sep 15 13:46:55.587934 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.587911    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5djd4" event={"ID":"9e72947c-265d-4fce-a5ad-4a8341bb14da","Type":"ContainerStarted","Data":"eaf2c1484227543c96c066c2aa194ca52497b9adaccc608a1538997998d56500"}
Sep 15 13:46:55.588035 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.587955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5djd4" event={"ID":"9e72947c-265d-4fce-a5ad-4a8341bb14da","Type":"ContainerStarted","Data":"3ca119c71d0b1473d70a68bd708a671e4bb8edbd15bffbe28ee8edfd9a9ac0f8"}
Sep 15 13:46:55.588035 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.587983    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5djd4"
Sep 15 13:46:55.589291 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.589271    2582 generic.go:356] "Generic (PLEG): container finished" podID="7e9be0ca-0577-400e-948a-8342573e7aa5" containerID="0082c307b7abf24559b14f1686553e18f4c677820209ef88a1b895eaaf16d202" exitCode=0
Sep 15 13:46:55.589648 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.589317    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:55.589648 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.589348    2582 scope.go:117] "RemoveContainer" containerID="0082c307b7abf24559b14f1686553e18f4c677820209ef88a1b895eaaf16d202"
Sep 15 13:46:55.590916 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.590897    2582 generic.go:356] "Generic (PLEG): container finished" podID="2df68655-9441-4325-a08c-7c5283226c85" containerID="b5ca5f6e18463950d103332dad382d9a5e8185efe5acf3c4f72881fe98bd18bb" exitCode=0
Sep 15 13:46:55.591036 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.590960    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:55.598005 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.597984    2582 scope.go:117] "RemoveContainer" containerID="0082c307b7abf24559b14f1686553e18f4c677820209ef88a1b895eaaf16d202"
Sep 15 13:46:55.598372 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:55.598349    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0082c307b7abf24559b14f1686553e18f4c677820209ef88a1b895eaaf16d202\": container with ID starting with 0082c307b7abf24559b14f1686553e18f4c677820209ef88a1b895eaaf16d202 not found: ID does not exist" containerID="0082c307b7abf24559b14f1686553e18f4c677820209ef88a1b895eaaf16d202"
Sep 15 13:46:55.598472 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.598378    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0082c307b7abf24559b14f1686553e18f4c677820209ef88a1b895eaaf16d202"} err="failed to get container status \"0082c307b7abf24559b14f1686553e18f4c677820209ef88a1b895eaaf16d202\": rpc error: code = NotFound desc = could not find container \"0082c307b7abf24559b14f1686553e18f4c677820209ef88a1b895eaaf16d202\": container with ID starting with 0082c307b7abf24559b14f1686553e18f4c677820209ef88a1b895eaaf16d202 not found: ID does not exist"
Sep 15 13:46:55.598472 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.598396    2582 scope.go:117] "RemoveContainer" containerID="b5ca5f6e18463950d103332dad382d9a5e8185efe5acf3c4f72881fe98bd18bb"
Sep 15 13:46:55.612249 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:55.612187    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-75lcg\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="7e9be0ca-0577-400e-948a-8342573e7aa5" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:55.612249 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.612198    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc" podStartSLOduration=1.612184557 podStartE2EDuration="1.612184557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:46:55.610495065 +0000 UTC m=+513.488397884" watchObservedRunningTime="2026-09-15 13:46:55.612184557 +0000 UTC m=+513.490087375"
Sep 15 13:46:55.624690 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.624668    2582 scope.go:117] "RemoveContainer" containerID="b5ca5f6e18463950d103332dad382d9a5e8185efe5acf3c4f72881fe98bd18bb"
Sep 15 13:46:55.625034 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:55.625000    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ca5f6e18463950d103332dad382d9a5e8185efe5acf3c4f72881fe98bd18bb\": container with ID starting with b5ca5f6e18463950d103332dad382d9a5e8185efe5acf3c4f72881fe98bd18bb not found: ID does not exist" containerID="b5ca5f6e18463950d103332dad382d9a5e8185efe5acf3c4f72881fe98bd18bb"
Sep 15 13:46:55.625128 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.625039    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ca5f6e18463950d103332dad382d9a5e8185efe5acf3c4f72881fe98bd18bb"} err="failed to get container status \"b5ca5f6e18463950d103332dad382d9a5e8185efe5acf3c4f72881fe98bd18bb\": rpc error: code = NotFound desc = could not find container \"b5ca5f6e18463950d103332dad382d9a5e8185efe5acf3c4f72881fe98bd18bb\": container with ID starting with b5ca5f6e18463950d103332dad382d9a5e8185efe5acf3c4f72881fe98bd18bb not found: ID does not exist"
Sep 15 13:46:55.636958 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.636915    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz" podStartSLOduration=1.6369016410000001 podStartE2EDuration="1.636901641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:46:55.635286151 +0000 UTC m=+513.513188995" watchObservedRunningTime="2026-09-15 13:46:55.636901641 +0000 UTC m=+513.514804461"
Sep 15 13:46:55.637089 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:55.637029    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-75lcg\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="7e9be0ca-0577-400e-948a-8342573e7aa5" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:55.654305 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:55.654254    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5djd4" podStartSLOduration=1.65424192 podStartE2EDuration="1.65424192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:46:55.653723077 +0000 UTC m=+513.531625896" watchObservedRunningTime="2026-09-15 13:46:55.65424192 +0000 UTC m=+513.532144737"
Sep 15 13:46:55.655307 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:55.655281    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="2df68655-9441-4325-a08c-7c5283226c85" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:55.656902 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:55.656880    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xjp9t\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="2df68655-9441-4325-a08c-7c5283226c85" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xjp9t"
Sep 15 13:46:55.658535 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:55.658516    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-75lcg\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="7e9be0ca-0577-400e-948a-8342573e7aa5" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-75lcg"
Sep 15 13:46:56.643911 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:56.643864    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2df68655-9441-4325-a08c-7c5283226c85" path="/var/lib/kubelet/pods/2df68655-9441-4325-a08c-7c5283226c85/volumes"
Sep 15 13:46:56.644328 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:56.644235    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e9be0ca-0577-400e-948a-8342573e7aa5" path="/var/lib/kubelet/pods/7e9be0ca-0577-400e-948a-8342573e7aa5/volumes"
Sep 15 13:46:57.236983 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:57.236928    2582 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-l5bvc_kuadrant-system_c3624b26-bc20-4a9b-9ca1-320889d71f54_0(f1df7e77d0b66a80814de0e2d4ffe3ec0c4ebac8c29e53d0a519dc45ea033bd4): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-l5bvc to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Sep 15 13:46:57.237055 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:57.237018    2582 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-l5bvc_kuadrant-system_c3624b26-bc20-4a9b-9ca1-320889d71f54_0(f1df7e77d0b66a80814de0e2d4ffe3ec0c4ebac8c29e53d0a519dc45ea033bd4): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-l5bvc to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:57.602776 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:57.602745    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:57.611026 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:57.611003    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:57.613248 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:57.613218    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-l5bvc\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="c3624b26-bc20-4a9b-9ca1-320889d71f54" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:57.714745 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:57.714696    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3624b26-bc20-4a9b-9ca1-320889d71f54-extensions-socket-volume\") pod \"c3624b26-bc20-4a9b-9ca1-320889d71f54\" (UID: \"c3624b26-bc20-4a9b-9ca1-320889d71f54\") "
Sep 15 13:46:57.715074 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:57.714802    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnf9s\" (UniqueName: \"kubernetes.io/projected/c3624b26-bc20-4a9b-9ca1-320889d71f54-kube-api-access-jnf9s\") pod \"c3624b26-bc20-4a9b-9ca1-320889d71f54\" (UID: \"c3624b26-bc20-4a9b-9ca1-320889d71f54\") "
Sep 15 13:46:57.715074 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:57.714967    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3624b26-bc20-4a9b-9ca1-320889d71f54-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c3624b26-bc20-4a9b-9ca1-320889d71f54" (UID: "c3624b26-bc20-4a9b-9ca1-320889d71f54"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:46:57.716897 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:57.716869    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3624b26-bc20-4a9b-9ca1-320889d71f54-kube-api-access-jnf9s" (OuterVolumeSpecName: "kube-api-access-jnf9s") pod "c3624b26-bc20-4a9b-9ca1-320889d71f54" (UID: "c3624b26-bc20-4a9b-9ca1-320889d71f54"). InnerVolumeSpecName "kube-api-access-jnf9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:46:57.816002 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:57.815957    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3624b26-bc20-4a9b-9ca1-320889d71f54-extensions-socket-volume\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:46:57.816002 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:57.815999    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnf9s\" (UniqueName: \"kubernetes.io/projected/c3624b26-bc20-4a9b-9ca1-320889d71f54-kube-api-access-jnf9s\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:46:58.133200 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:58.133165    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:58.197112 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:58.197077    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:58.607618 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:58.607586    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:58.609694 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:58.609667    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-l5bvc\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="c3624b26-bc20-4a9b-9ca1-320889d71f54" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:58.615102 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:46:58.615076    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-l5bvc\" is forbidden: User \"system:node:ip-10-0-138-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-252.ec2.internal' and this object" podUID="c3624b26-bc20-4a9b-9ca1-320889d71f54" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-l5bvc"
Sep 15 13:46:58.642451 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:58.642422    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3624b26-bc20-4a9b-9ca1-320889d71f54" path="/var/lib/kubelet/pods/c3624b26-bc20-4a9b-9ca1-320889d71f54/volumes"
Sep 15 13:46:58.740121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:58.740086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:46:58.864026 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:46:58.863935    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:03.939731 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:03.939690    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:04.039804 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:04.039767    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:06.601522 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:06.601492    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"
Sep 15 13:47:06.601881 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:06.601567    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"
Sep 15 13:47:06.601947 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:06.601926    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5djd4"
Sep 15 13:47:06.675375 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:06.675344    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"]
Sep 15 13:47:06.675557 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:06.675525    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz" podUID="7af63032-a064-448d-865c-7cfdab1cb6c2" containerName="manager" containerID="cri-o://4e3a0bd541abc82898f0a48fcc6d99099b0728812140e330a3a565223931cdab" gracePeriod=10
Sep 15 13:47:06.874021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:06.873933    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6"]
Sep 15 13:47:06.879387 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:06.879364    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6"
Sep 15 13:47:06.893771 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:06.893737    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6"]
Sep 15 13:47:06.945009 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:06.944983    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"
Sep 15 13:47:06.973812 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:06.973778    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:06.996747 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:06.996717    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-przzs\" (UniqueName: \"kubernetes.io/projected/d5dd8efd-326c-441c-b845-0a08204e01c3-kube-api-access-przzs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2cbx6\" (UID: \"d5dd8efd-326c-441c-b845-0a08204e01c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6"
Sep 15 13:47:06.996894 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:06.996759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8efd-326c-441c-b845-0a08204e01c3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2cbx6\" (UID: \"d5dd8efd-326c-441c-b845-0a08204e01c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6"
Sep 15 13:47:07.068596 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.068562    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:07.097402 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.097374    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7af63032-a064-448d-865c-7cfdab1cb6c2-extensions-socket-volume\") pod \"7af63032-a064-448d-865c-7cfdab1cb6c2\" (UID: \"7af63032-a064-448d-865c-7cfdab1cb6c2\") "
Sep 15 13:47:07.097515 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.097420    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48zmm\" (UniqueName: \"kubernetes.io/projected/7af63032-a064-448d-865c-7cfdab1cb6c2-kube-api-access-48zmm\") pod \"7af63032-a064-448d-865c-7cfdab1cb6c2\" (UID: \"7af63032-a064-448d-865c-7cfdab1cb6c2\") "
Sep 15 13:47:07.097551 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.097536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-przzs\" (UniqueName: \"kubernetes.io/projected/d5dd8efd-326c-441c-b845-0a08204e01c3-kube-api-access-przzs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2cbx6\" (UID: \"d5dd8efd-326c-441c-b845-0a08204e01c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6"
Sep 15 13:47:07.097592 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.097567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8efd-326c-441c-b845-0a08204e01c3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2cbx6\" (UID: \"d5dd8efd-326c-441c-b845-0a08204e01c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6"
Sep 15 13:47:07.097768 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.097744    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af63032-a064-448d-865c-7cfdab1cb6c2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7af63032-a064-448d-865c-7cfdab1cb6c2" (UID: "7af63032-a064-448d-865c-7cfdab1cb6c2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:47:07.097940 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.097922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5dd8efd-326c-441c-b845-0a08204e01c3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2cbx6\" (UID: \"d5dd8efd-326c-441c-b845-0a08204e01c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6"
Sep 15 13:47:07.099511 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.099488    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af63032-a064-448d-865c-7cfdab1cb6c2-kube-api-access-48zmm" (OuterVolumeSpecName: "kube-api-access-48zmm") pod "7af63032-a064-448d-865c-7cfdab1cb6c2" (UID: "7af63032-a064-448d-865c-7cfdab1cb6c2"). InnerVolumeSpecName "kube-api-access-48zmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:47:07.109609 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.109588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-przzs\" (UniqueName: \"kubernetes.io/projected/d5dd8efd-326c-441c-b845-0a08204e01c3-kube-api-access-przzs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2cbx6\" (UID: \"d5dd8efd-326c-441c-b845-0a08204e01c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6"
Sep 15 13:47:07.192463 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.192377    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6"
Sep 15 13:47:07.198689 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.198659    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48zmm\" (UniqueName: \"kubernetes.io/projected/7af63032-a064-448d-865c-7cfdab1cb6c2-kube-api-access-48zmm\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:47:07.198689 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.198684    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7af63032-a064-448d-865c-7cfdab1cb6c2-extensions-socket-volume\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:47:07.495310 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.495226    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6"]
Sep 15 13:47:07.498830 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:47:07.498805    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dd8efd_326c_441c_b845_0a08204e01c3.slice/crio-98a77e56befd482c859791d69377ec18b0e7d9e395eeccfe5f26fcae6f112c54 WatchSource:0}: Error finding container 98a77e56befd482c859791d69377ec18b0e7d9e395eeccfe5f26fcae6f112c54: Status 404 returned error can't find the container with id 98a77e56befd482c859791d69377ec18b0e7d9e395eeccfe5f26fcae6f112c54
Sep 15 13:47:07.651783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.651741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6" event={"ID":"d5dd8efd-326c-441c-b845-0a08204e01c3","Type":"ContainerStarted","Data":"0d1717109196549b4eb6bd990afaa01f273d7d090c1fc8f6f6aa0eb8d2f0bfae"}
Sep 15 13:47:07.652228 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.651802    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6" event={"ID":"d5dd8efd-326c-441c-b845-0a08204e01c3","Type":"ContainerStarted","Data":"98a77e56befd482c859791d69377ec18b0e7d9e395eeccfe5f26fcae6f112c54"}
Sep 15 13:47:07.652228 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.651827    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6"
Sep 15 13:47:07.653195 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.653168    2582 generic.go:356] "Generic (PLEG): container finished" podID="7af63032-a064-448d-865c-7cfdab1cb6c2" containerID="4e3a0bd541abc82898f0a48fcc6d99099b0728812140e330a3a565223931cdab" exitCode=0
Sep 15 13:47:07.653293 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.653200    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"
Sep 15 13:47:07.653293 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.653249    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz" event={"ID":"7af63032-a064-448d-865c-7cfdab1cb6c2","Type":"ContainerDied","Data":"4e3a0bd541abc82898f0a48fcc6d99099b0728812140e330a3a565223931cdab"}
Sep 15 13:47:07.653422 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.653311    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz" event={"ID":"7af63032-a064-448d-865c-7cfdab1cb6c2","Type":"ContainerDied","Data":"aa233353d9ffed9b65572ff736dd2b8bdfdf4342b30774f58ca661bf64a00f9e"}
Sep 15 13:47:07.653422 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.653347    2582 scope.go:117] "RemoveContainer" containerID="4e3a0bd541abc82898f0a48fcc6d99099b0728812140e330a3a565223931cdab"
Sep 15 13:47:07.663947 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.663912    2582 scope.go:117] "RemoveContainer" containerID="4e3a0bd541abc82898f0a48fcc6d99099b0728812140e330a3a565223931cdab"
Sep 15 13:47:07.664662 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:47:07.664635    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3a0bd541abc82898f0a48fcc6d99099b0728812140e330a3a565223931cdab\": container with ID starting with 4e3a0bd541abc82898f0a48fcc6d99099b0728812140e330a3a565223931cdab not found: ID does not exist" containerID="4e3a0bd541abc82898f0a48fcc6d99099b0728812140e330a3a565223931cdab"
Sep 15 13:47:07.664751 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.664671    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3a0bd541abc82898f0a48fcc6d99099b0728812140e330a3a565223931cdab"} err="failed to get container status \"4e3a0bd541abc82898f0a48fcc6d99099b0728812140e330a3a565223931cdab\": rpc error: code = NotFound desc = could not find container \"4e3a0bd541abc82898f0a48fcc6d99099b0728812140e330a3a565223931cdab\": container with ID starting with 4e3a0bd541abc82898f0a48fcc6d99099b0728812140e330a3a565223931cdab not found: ID does not exist"
Sep 15 13:47:07.670998 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.670953    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6" podStartSLOduration=1.670938837 podStartE2EDuration="1.670938837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:47:07.669798639 +0000 UTC m=+525.547701460" watchObservedRunningTime="2026-09-15 13:47:07.670938837 +0000 UTC m=+525.548841656"
Sep 15 13:47:07.686104 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.686082    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"]
Sep 15 13:47:07.692118 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:07.692095    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-54jdz"]
Sep 15 13:47:08.643314 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:08.643281    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af63032-a064-448d-865c-7cfdab1cb6c2" path="/var/lib/kubelet/pods/7af63032-a064-448d-865c-7cfdab1cb6c2/volumes"
Sep 15 13:47:12.073065 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:12.073026    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:12.170837 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:12.170797    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:12.271255 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:12.271215    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:18.661429 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:18.661356    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2cbx6"
Sep 15 13:47:18.714032 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:18.714004    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"]
Sep 15 13:47:18.714226 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:18.714206    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc" podUID="fdc88d72-d3d2-41d7-a03e-009283733208" containerName="manager" containerID="cri-o://727ff7d3f3ffb04b4496fb4a1f3d6cd1ddd93110c41898d48c22109c8172fbf6" gracePeriod=10
Sep 15 13:47:18.968309 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:18.968285    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"
Sep 15 13:47:18.984303 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:18.984279    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p528m\" (UniqueName: \"kubernetes.io/projected/fdc88d72-d3d2-41d7-a03e-009283733208-kube-api-access-p528m\") pod \"fdc88d72-d3d2-41d7-a03e-009283733208\" (UID: \"fdc88d72-d3d2-41d7-a03e-009283733208\") "
Sep 15 13:47:18.984389 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:18.984332    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc88d72-d3d2-41d7-a03e-009283733208-extensions-socket-volume\") pod \"fdc88d72-d3d2-41d7-a03e-009283733208\" (UID: \"fdc88d72-d3d2-41d7-a03e-009283733208\") "
Sep 15 13:47:18.984703 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:18.984678    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc88d72-d3d2-41d7-a03e-009283733208-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fdc88d72-d3d2-41d7-a03e-009283733208" (UID: "fdc88d72-d3d2-41d7-a03e-009283733208"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:47:18.986166 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:18.986150    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc88d72-d3d2-41d7-a03e-009283733208-kube-api-access-p528m" (OuterVolumeSpecName: "kube-api-access-p528m") pod "fdc88d72-d3d2-41d7-a03e-009283733208" (UID: "fdc88d72-d3d2-41d7-a03e-009283733208"). InnerVolumeSpecName "kube-api-access-p528m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:47:19.085269 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:19.085229    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc88d72-d3d2-41d7-a03e-009283733208-extensions-socket-volume\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:47:19.085269 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:19.085260    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p528m\" (UniqueName: \"kubernetes.io/projected/fdc88d72-d3d2-41d7-a03e-009283733208-kube-api-access-p528m\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:47:19.710227 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:19.710191    2582 generic.go:356] "Generic (PLEG): container finished" podID="fdc88d72-d3d2-41d7-a03e-009283733208" containerID="727ff7d3f3ffb04b4496fb4a1f3d6cd1ddd93110c41898d48c22109c8172fbf6" exitCode=0
Sep 15 13:47:19.710694 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:19.710245    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"
Sep 15 13:47:19.710694 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:19.710271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc" event={"ID":"fdc88d72-d3d2-41d7-a03e-009283733208","Type":"ContainerDied","Data":"727ff7d3f3ffb04b4496fb4a1f3d6cd1ddd93110c41898d48c22109c8172fbf6"}
Sep 15 13:47:19.710694 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:19.710322    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc" event={"ID":"fdc88d72-d3d2-41d7-a03e-009283733208","Type":"ContainerDied","Data":"60c0c6843d3602b68413319eb1530bf991b7edf8f2543c6217025e37bcbd23de"}
Sep 15 13:47:19.710694 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:19.710345    2582 scope.go:117] "RemoveContainer" containerID="727ff7d3f3ffb04b4496fb4a1f3d6cd1ddd93110c41898d48c22109c8172fbf6"
Sep 15 13:47:19.718790 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:19.718578    2582 scope.go:117] "RemoveContainer" containerID="727ff7d3f3ffb04b4496fb4a1f3d6cd1ddd93110c41898d48c22109c8172fbf6"
Sep 15 13:47:19.718867 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:47:19.718849    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727ff7d3f3ffb04b4496fb4a1f3d6cd1ddd93110c41898d48c22109c8172fbf6\": container with ID starting with 727ff7d3f3ffb04b4496fb4a1f3d6cd1ddd93110c41898d48c22109c8172fbf6 not found: ID does not exist" containerID="727ff7d3f3ffb04b4496fb4a1f3d6cd1ddd93110c41898d48c22109c8172fbf6"
Sep 15 13:47:19.718906 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:19.718874    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727ff7d3f3ffb04b4496fb4a1f3d6cd1ddd93110c41898d48c22109c8172fbf6"} err="failed to get container status \"727ff7d3f3ffb04b4496fb4a1f3d6cd1ddd93110c41898d48c22109c8172fbf6\": rpc error: code = NotFound desc = could not find container \"727ff7d3f3ffb04b4496fb4a1f3d6cd1ddd93110c41898d48c22109c8172fbf6\": container with ID starting with 727ff7d3f3ffb04b4496fb4a1f3d6cd1ddd93110c41898d48c22109c8172fbf6 not found: ID does not exist"
Sep 15 13:47:19.741991 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:19.738479    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"]
Sep 15 13:47:19.744067 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:19.744032    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-qn5rc"]
Sep 15 13:47:20.643217 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:20.643181    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdc88d72-d3d2-41d7-a03e-009283733208" path="/var/lib/kubelet/pods/fdc88d72-d3d2-41d7-a03e-009283733208/volumes"
Sep 15 13:47:20.978909 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:20.978811    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:21.181204 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:21.181169    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:22.985243 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:22.985211    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"]
Sep 15 13:47:22.985619 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:22.985512    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc88d72-d3d2-41d7-a03e-009283733208" containerName="manager"
Sep 15 13:47:22.985619 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:22.985523    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc88d72-d3d2-41d7-a03e-009283733208" containerName="manager"
Sep 15 13:47:22.985619 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:22.985536    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af63032-a064-448d-865c-7cfdab1cb6c2" containerName="manager"
Sep 15 13:47:22.985619 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:22.985541    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af63032-a064-448d-865c-7cfdab1cb6c2" containerName="manager"
Sep 15 13:47:22.985619 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:22.985621    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af63032-a064-448d-865c-7cfdab1cb6c2" containerName="manager"
Sep 15 13:47:22.985798 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:22.985628    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc88d72-d3d2-41d7-a03e-009283733208" containerName="manager"
Sep 15 13:47:22.988272 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:22.988251    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:22.990546 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:22.990525    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 13:47:22.990685 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:22.990631    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qmb7r\""
Sep 15 13:47:23.001073 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.001036    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"]
Sep 15 13:47:23.017934 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.017900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.018080 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.017949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623e2217-eca0-464b-b50b-86e331bf2a7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.018080 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.017964    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvxk\" (UniqueName: \"kubernetes.io/projected/623e2217-eca0-464b-b50b-86e331bf2a7e-kube-api-access-2pvxk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.018080 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.018028    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.018080 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.018071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.018315 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.018101    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.018315 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.018118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623e2217-eca0-464b-b50b-86e331bf2a7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.018315 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.018151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623e2217-eca0-464b-b50b-86e331bf2a7e-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.018315 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.018198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.022960 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.022937    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:23.119024 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.118985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623e2217-eca0-464b-b50b-86e331bf2a7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.119024 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.119024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvxk\" (UniqueName: \"kubernetes.io/projected/623e2217-eca0-464b-b50b-86e331bf2a7e-kube-api-access-2pvxk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.119317 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.119042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.119317 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.119113    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.119317 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.119273    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.119478 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.119316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623e2217-eca0-464b-b50b-86e331bf2a7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.119478 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.119353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623e2217-eca0-464b-b50b-86e331bf2a7e-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.119478 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.119389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.119478 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.119418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.119678 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.119534    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.119798 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.119774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.119880 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.119853    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623e2217-eca0-464b-b50b-86e331bf2a7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.119938 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.119890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.120000 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.119952    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.121547 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.121524    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623e2217-eca0-464b-b50b-86e331bf2a7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.121734 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.121717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623e2217-eca0-464b-b50b-86e331bf2a7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.126539 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.126512    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623e2217-eca0-464b-b50b-86e331bf2a7e-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.126732 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.126710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvxk\" (UniqueName: \"kubernetes.io/projected/623e2217-eca0-464b-b50b-86e331bf2a7e-kube-api-access-2pvxk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-n4f4m\" (UID: \"623e2217-eca0-464b-b50b-86e331bf2a7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.302087 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.301993    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:23.523643 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.523595    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:23.575075 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.575050    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"]
Sep 15 13:47:23.576451 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:47:23.576426    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623e2217_eca0_464b_b50b_86e331bf2a7e.slice/crio-5663cd51d4f6fce0a80fd5d95fba8d27699442123c942d1233db4e33d9a06fa0 WatchSource:0}: Error finding container 5663cd51d4f6fce0a80fd5d95fba8d27699442123c942d1233db4e33d9a06fa0: Status 404 returned error can't find the container with id 5663cd51d4f6fce0a80fd5d95fba8d27699442123c942d1233db4e33d9a06fa0
Sep 15 13:47:23.729967 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:23.729937    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m" event={"ID":"623e2217-eca0-464b-b50b-86e331bf2a7e","Type":"ContainerStarted","Data":"5663cd51d4f6fce0a80fd5d95fba8d27699442123c942d1233db4e33d9a06fa0"}
Sep 15 13:47:28.881487 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:28.881448    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:47:28.881788 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:28.881528    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:47:28.881788 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:28.881576    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:47:29.760021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:29.759990    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m" event={"ID":"623e2217-eca0-464b-b50b-86e331bf2a7e","Type":"ContainerStarted","Data":"f007502291d71235dd9a4e4087317ecfa7c569224889c8dcff0ca4ec9517f604"}
Sep 15 13:47:29.780250 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:29.780205    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m" podStartSLOduration=2.477496725 podStartE2EDuration="7.780189693s" totalImagesPullingTime="5.302692968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:47:29.778862734 +0000 UTC m=+547.656765577" watchObservedRunningTime="2026-09-15 13:47:29.780189693 +0000 UTC m=+547.658092512"
Sep 15 13:47:30.302377 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:30.302333    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:30.307182 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:30.307134    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:30.764108 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:30.764081    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:30.765009 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:30.764986    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:47:36.015160 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.015098    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:36.119431 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.119400    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:36.174001 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.173964    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:36.492570 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.492531    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:47:36.496218 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.496196    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-m4784"
Sep 15 13:47:36.498293 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.498269    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 13:47:36.504067 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.504041    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:47:36.527733 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.527706    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:47:36.627540 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.627508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8v5s\" (UniqueName: \"kubernetes.io/projected/8c40ccfe-b343-46d8-86d1-840e9af8f4fc-kube-api-access-v8v5s\") pod \"limitador-limitador-84994f7ffc-m4784\" (UID: \"8c40ccfe-b343-46d8-86d1-840e9af8f4fc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-m4784"
Sep 15 13:47:36.627716 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.627548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c40ccfe-b343-46d8-86d1-840e9af8f4fc-config-file\") pod \"limitador-limitador-84994f7ffc-m4784\" (UID: \"8c40ccfe-b343-46d8-86d1-840e9af8f4fc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-m4784"
Sep 15 13:47:36.728652 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.728617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8v5s\" (UniqueName: \"kubernetes.io/projected/8c40ccfe-b343-46d8-86d1-840e9af8f4fc-kube-api-access-v8v5s\") pod \"limitador-limitador-84994f7ffc-m4784\" (UID: \"8c40ccfe-b343-46d8-86d1-840e9af8f4fc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-m4784"
Sep 15 13:47:36.728830 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.728675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c40ccfe-b343-46d8-86d1-840e9af8f4fc-config-file\") pod \"limitador-limitador-84994f7ffc-m4784\" (UID: \"8c40ccfe-b343-46d8-86d1-840e9af8f4fc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-m4784"
Sep 15 13:47:36.729356 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.729335    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c40ccfe-b343-46d8-86d1-840e9af8f4fc-config-file\") pod \"limitador-limitador-84994f7ffc-m4784\" (UID: \"8c40ccfe-b343-46d8-86d1-840e9af8f4fc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-m4784"
Sep 15 13:47:36.737233 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.737207    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8v5s\" (UniqueName: \"kubernetes.io/projected/8c40ccfe-b343-46d8-86d1-840e9af8f4fc-kube-api-access-v8v5s\") pod \"limitador-limitador-84994f7ffc-m4784\" (UID: \"8c40ccfe-b343-46d8-86d1-840e9af8f4fc\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-m4784"
Sep 15 13:47:36.806677 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.806594    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-m4784"
Sep 15 13:47:36.834313 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.834272    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:36.935205 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:36.935173    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:37.077510 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:37.077485    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:47:37.080029 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:47:37.080004    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c40ccfe_b343_46d8_86d1_840e9af8f4fc.slice/crio-feee850893d9881b8f696a67693438635e71df3f1d9aad7c44a7404997457de2 WatchSource:0}: Error finding container feee850893d9881b8f696a67693438635e71df3f1d9aad7c44a7404997457de2: Status 404 returned error can't find the container with id feee850893d9881b8f696a67693438635e71df3f1d9aad7c44a7404997457de2
Sep 15 13:47:37.797405 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:37.797369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-m4784" event={"ID":"8c40ccfe-b343-46d8-86d1-840e9af8f4fc","Type":"ContainerStarted","Data":"feee850893d9881b8f696a67693438635e71df3f1d9aad7c44a7404997457de2"}
Sep 15 13:47:38.803552 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:38.803468    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-m4784" event={"ID":"8c40ccfe-b343-46d8-86d1-840e9af8f4fc","Type":"ContainerStarted","Data":"b11f21c506c01b88dc61f74b56a9b51ea9aff3f036cc107f769d647a3ecddc9d"}
Sep 15 13:47:38.803916 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:38.803690    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-m4784"
Sep 15 13:47:38.820990 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:38.820939    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-m4784" podStartSLOduration=1.484085734 podStartE2EDuration="2.820924675s" totalImagesPullingTime="1.336838941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:47:38.819865955 +0000 UTC m=+556.697768773" watchObservedRunningTime="2026-09-15 13:47:38.820924675 +0000 UTC m=+556.698827494"
Sep 15 13:47:41.427934 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:41.427897    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:41.529489 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:41.529454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:49.809084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:49.809054    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-m4784"
Sep 15 13:47:51.955447 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:51.955406    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:52.053073 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:52.053037    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:47:52.454152 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:47:52.454105    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:05.221858 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:05.221821    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:06.387751 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:06.387717    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:12.732549 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:12.732511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:12.877530 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:12.877493    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:12.986731 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:12.986647    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:13.090545 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:13.090513    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:13.152022 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:13.151989    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:16.399497 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:16.399436    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:16.499086 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:16.499044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:37.557835 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:37.557795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:37.659134 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:37.659097    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:47.301652 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:47.301570    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:47.392412 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:47.392371    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:48.709359 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:48.709310    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:48.829976 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:48.829937    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:48.976709 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:48.976631    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:49.268936 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:49.268837    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:49.431296 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:49.431259    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:49.526618 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:49.526533    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:48:50.127664 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:50.127630    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:48:59.156540 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:59.156460    2582 generic.go:356] "Generic (PLEG): container finished" podID="623e2217-eca0-464b-b50b-86e331bf2a7e" containerID="f007502291d71235dd9a4e4087317ecfa7c569224889c8dcff0ca4ec9517f604" exitCode=0
Sep 15 13:48:59.156916 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:59.156533    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m" event={"ID":"623e2217-eca0-464b-b50b-86e331bf2a7e","Type":"ContainerDied","Data":"f007502291d71235dd9a4e4087317ecfa7c569224889c8dcff0ca4ec9517f604"}
Sep 15 13:48:59.157027 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:59.157007    2582 scope.go:117] "RemoveContainer" containerID="f007502291d71235dd9a4e4087317ecfa7c569224889c8dcff0ca4ec9517f604"
Sep 15 13:48:59.158265 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:59.158230    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:48:59.158363 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:59.158310    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:48:59.158363 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:48:59.158341    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:49:00.163534 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:00.163484    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m" event={"ID":"623e2217-eca0-464b-b50b-86e331bf2a7e","Type":"ContainerStarted","Data":"ae769d258a6dab9392ed1320e9ab46c2151b5aa2db87c27763d2fd9c4d2fa764"}
Sep 15 13:49:00.302509 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:00.302476    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:49:00.303732 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:00.303703    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-n4f4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 15 13:49:00.303844 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:00.303750    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m" podUID="623e2217-eca0-464b-b50b-86e331bf2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 15 13:49:00.765285 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:00.765249    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:49:01.146509 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:01.146470    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:49:01.269930 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:01.269892    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:49:01.303303 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:01.303269    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-n4f4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 15 13:49:01.303460 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:01.303324    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m" podUID="623e2217-eca0-464b-b50b-86e331bf2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 15 13:49:02.302487 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:02.302449    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-n4f4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 15 13:49:02.302871 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:02.302505    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m" podUID="623e2217-eca0-464b-b50b-86e331bf2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 15 13:49:03.302799 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:03.302760    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-n4f4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 15 13:49:03.303220 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:03.302817    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m" podUID="623e2217-eca0-464b-b50b-86e331bf2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 15 13:49:04.303376 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:04.303340    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-n4f4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 15 13:49:04.303762 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:04.303399    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m" podUID="623e2217-eca0-464b-b50b-86e331bf2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 15 13:49:05.303344 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:05.303300    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-n4f4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 15 13:49:05.303522 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:05.303373    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m" podUID="623e2217-eca0-464b-b50b-86e331bf2a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 15 13:49:06.306758 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:06.306727    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:49:06.307864 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:06.307840    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-n4f4m"
Sep 15 13:49:11.352955 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:11.352920    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:49:11.923909 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:11.923873    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:49:25.866820 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:25.866781    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:49:34.358488 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:34.358453    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:49:35.296625 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:35.296582    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:49:35.995936 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:35.995896    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:49:38.524010 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.523978    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"]
Sep 15 13:49:38.526357 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.526334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.537397 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.537370    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5mmvf\""
Sep 15 13:49:38.537522 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.537391    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 13:49:38.537522 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.537418    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 13:49:38.537522 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.537439    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 15 13:49:38.540709 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.540687    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"]
Sep 15 13:49:38.563292 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.563255    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:49:38.610264 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.610227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/048f743b-9f78-4129-8c47-e77820f42f68-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.610264 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.610268    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/048f743b-9f78-4129-8c47-e77820f42f68-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.610495 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.610288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/048f743b-9f78-4129-8c47-e77820f42f68-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.610495 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.610306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/048f743b-9f78-4129-8c47-e77820f42f68-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.610495 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.610424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm944\" (UniqueName: \"kubernetes.io/projected/048f743b-9f78-4129-8c47-e77820f42f68-kube-api-access-nm944\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.610495 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.610487    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048f743b-9f78-4129-8c47-e77820f42f68-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.710854 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.710813    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm944\" (UniqueName: \"kubernetes.io/projected/048f743b-9f78-4129-8c47-e77820f42f68-kube-api-access-nm944\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.710854 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.710858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048f743b-9f78-4129-8c47-e77820f42f68-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.711123 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.710889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/048f743b-9f78-4129-8c47-e77820f42f68-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.711123 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.710905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/048f743b-9f78-4129-8c47-e77820f42f68-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.711123 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.710921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/048f743b-9f78-4129-8c47-e77820f42f68-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.711123 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.710939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/048f743b-9f78-4129-8c47-e77820f42f68-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.711389 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.711359    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/048f743b-9f78-4129-8c47-e77820f42f68-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.711451 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.711388    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048f743b-9f78-4129-8c47-e77820f42f68-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.711451 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.711423    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/048f743b-9f78-4129-8c47-e77820f42f68-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.713357 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.713334    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/048f743b-9f78-4129-8c47-e77820f42f68-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.713592 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.713570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/048f743b-9f78-4129-8c47-e77820f42f68-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.723257 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.723230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm944\" (UniqueName: \"kubernetes.io/projected/048f743b-9f78-4129-8c47-e77820f42f68-kube-api-access-nm944\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk\" (UID: \"048f743b-9f78-4129-8c47-e77820f42f68\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:38.836857 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:38.836832    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:39.108363 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:39.108333    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"]
Sep 15 13:49:39.110756 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:49:39.110728    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048f743b_9f78_4129_8c47_e77820f42f68.slice/crio-3ec7f1ce75fcc381b6c9f8db5dcd0e602556045b626f7508c649c194705e42a1 WatchSource:0}: Error finding container 3ec7f1ce75fcc381b6c9f8db5dcd0e602556045b626f7508c649c194705e42a1: Status 404 returned error can't find the container with id 3ec7f1ce75fcc381b6c9f8db5dcd0e602556045b626f7508c649c194705e42a1
Sep 15 13:49:39.112057 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:39.112034    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 13:49:39.352092 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:39.352048    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk" event={"ID":"048f743b-9f78-4129-8c47-e77820f42f68","Type":"ContainerStarted","Data":"3ec7f1ce75fcc381b6c9f8db5dcd0e602556045b626f7508c649c194705e42a1"}
Sep 15 13:49:39.736637 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:39.736603    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:49:40.359937 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:40.359902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk" event={"ID":"048f743b-9f78-4129-8c47-e77820f42f68","Type":"ContainerStarted","Data":"50a488f7190dec8e2fccddb28c6b1e833ff8984355406ba824dd7fb437e3bf9e"}
Sep 15 13:49:40.397566 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:40.397511    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk" podStartSLOduration=1.427232775 podStartE2EDuration="2.397495915s" totalImagesPullingTime="970.26314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:49:40.396397738 +0000 UTC m=+678.274300558" watchObservedRunningTime="2026-09-15 13:49:40.397495915 +0000 UTC m=+678.275398733"
Sep 15 13:49:48.532733 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.532684    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"]
Sep 15 13:49:48.535960 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.535937    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.538338 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.538317    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 15 13:49:48.554845 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.554815    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"]
Sep 15 13:49:48.563361 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.563327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:49:48.601544 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.601510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5lvd\" (UniqueName: \"kubernetes.io/projected/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-kube-api-access-d5lvd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.601687 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.601560    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.601687 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.601631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.601687 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.601653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.601687 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.601680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.601860 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.601702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.702375 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.702333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.702563 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.702391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5lvd\" (UniqueName: \"kubernetes.io/projected/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-kube-api-access-d5lvd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.702563 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.702433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.702711 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.702682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.702778 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.702762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.702944 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.702890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.702944 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.702897    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.703135 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.703113    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.703225 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.703173    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.704751 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.704731    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.705255 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.705234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.710761 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.710736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5lvd\" (UniqueName: \"kubernetes.io/projected/3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa-kube-api-access-d5lvd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-st4nc\" (UID: \"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.837990 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.837951    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:48.838300 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.838284    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:48.846948 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.846921    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:48.850747 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:48.850727    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:49.118843 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:49.118766    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"]
Sep 15 13:49:49.122664 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:49:49.122636    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5138aa_cc8f_4b5e_bdae_3e4750a10aaa.slice/crio-689338083d3f2193feb74d6f41845404ac531363722b3f4bb63c1eba612f74db WatchSource:0}: Error finding container 689338083d3f2193feb74d6f41845404ac531363722b3f4bb63c1eba612f74db: Status 404 returned error can't find the container with id 689338083d3f2193feb74d6f41845404ac531363722b3f4bb63c1eba612f74db
Sep 15 13:49:49.402596 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:49.402501    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc" event={"ID":"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa","Type":"ContainerStarted","Data":"7e5a013fd2f2dcff400c47a3857f52bb43831845fa3240469fe99b888c696771"}
Sep 15 13:49:49.402596 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:49.402568    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc" event={"ID":"3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa","Type":"ContainerStarted","Data":"689338083d3f2193feb74d6f41845404ac531363722b3f4bb63c1eba612f74db"}
Sep 15 13:49:49.413765 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:49.413741    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk"
Sep 15 13:49:49.422941 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:49.422888    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc" podStartSLOduration=1.246273932 podStartE2EDuration="1.422869992s" totalImagesPullingTime="176.59606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:49:49.420892111 +0000 UTC m=+687.298794931" watchObservedRunningTime="2026-09-15 13:49:49.422869992 +0000 UTC m=+687.300772813"
Sep 15 13:49:49.819131 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:49.819056    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:49:58.848193 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:58.848151    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:58.848637 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:58.848220    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:58.860608 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:58.860580    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:58.906784 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:58.906750    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"]
Sep 15 13:49:58.911255 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:58.911231    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:58.913724 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:58.913703    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 15 13:49:58.919524 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:58.919500    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"]
Sep 15 13:49:58.988129 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:58.988080    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99775774-70c0-4023-a283-611d584af76b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:58.988339 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:58.988179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99775774-70c0-4023-a283-611d584af76b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:58.988339 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:58.988204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99775774-70c0-4023-a283-611d584af76b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:58.988339 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:58.988228    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99775774-70c0-4023-a283-611d584af76b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:58.988339 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:58.988312    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99775774-70c0-4023-a283-611d584af76b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:58.988571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:58.988361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hpw2\" (UniqueName: \"kubernetes.io/projected/99775774-70c0-4023-a283-611d584af76b-kube-api-access-5hpw2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.015216 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.015182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:49:59.089760 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.089728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99775774-70c0-4023-a283-611d584af76b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.089922 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.089776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99775774-70c0-4023-a283-611d584af76b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.089922 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.089797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99775774-70c0-4023-a283-611d584af76b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.089922 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.089828    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99775774-70c0-4023-a283-611d584af76b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.089922 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.089865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99775774-70c0-4023-a283-611d584af76b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.089922 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.089894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hpw2\" (UniqueName: \"kubernetes.io/projected/99775774-70c0-4023-a283-611d584af76b-kube-api-access-5hpw2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.090218 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.090126    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99775774-70c0-4023-a283-611d584af76b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.090289 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.090257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99775774-70c0-4023-a283-611d584af76b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.090289 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.090281    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99775774-70c0-4023-a283-611d584af76b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.092111 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.092090    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99775774-70c0-4023-a283-611d584af76b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.092191 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.092176    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99775774-70c0-4023-a283-611d584af76b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.099409 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.099352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hpw2\" (UniqueName: \"kubernetes.io/projected/99775774-70c0-4023-a283-611d584af76b-kube-api-access-5hpw2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54\" (UID: \"99775774-70c0-4023-a283-611d584af76b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.223452 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.223414    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:49:59.459349 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.459275    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-st4nc"
Sep 15 13:49:59.499079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:49:59.499052    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"]
Sep 15 13:49:59.500644 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:49:59.500618    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99775774_70c0_4023_a283_611d584af76b.slice/crio-70bd78c8bfbf449644d4a59a039e344d71d4a67f92d9ada0ff659a75536b881c WatchSource:0}: Error finding container 70bd78c8bfbf449644d4a59a039e344d71d4a67f92d9ada0ff659a75536b881c: Status 404 returned error can't find the container with id 70bd78c8bfbf449644d4a59a039e344d71d4a67f92d9ada0ff659a75536b881c
Sep 15 13:50:00.423915 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:00.423877    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:50:00.454633 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:00.454595    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54" event={"ID":"99775774-70c0-4023-a283-611d584af76b","Type":"ContainerStarted","Data":"0f359f7398e16ef0d0edbeb636c0e87a389850d4e6fccaad04c9c22ffa76f79b"}
Sep 15 13:50:00.454790 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:00.454652    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54" event={"ID":"99775774-70c0-4023-a283-611d584af76b","Type":"ContainerStarted","Data":"70bd78c8bfbf449644d4a59a039e344d71d4a67f92d9ada0ff659a75536b881c"}
Sep 15 13:50:00.473720 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:00.473656    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54" podStartSLOduration=2.251457346 podStartE2EDuration="2.47363462s" totalImagesPullingTime="222.177274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:50:00.47115496 +0000 UTC m=+698.349057772" watchObservedRunningTime="2026-09-15 13:50:00.47363462 +0000 UTC m=+698.351537440"
Sep 15 13:50:04.222554 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.221029    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"]
Sep 15 13:50:04.227314 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.227285    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.229827 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.229806    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 15 13:50:04.236830 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.236801    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"]
Sep 15 13:50:04.256095 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.256057    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:50:04.339122 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.339084    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ca03da-589d-49f1-825e-f5b22caf2b26-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.339122 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.339120    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ca03da-589d-49f1-825e-f5b22caf2b26-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.339361 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.339172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ca03da-589d-49f1-825e-f5b22caf2b26-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.339361 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.339230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ca03da-589d-49f1-825e-f5b22caf2b26-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.339361 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.339268    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ca03da-589d-49f1-825e-f5b22caf2b26-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.339361 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.339298    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvsvl\" (UniqueName: \"kubernetes.io/projected/20ca03da-589d-49f1-825e-f5b22caf2b26-kube-api-access-vvsvl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.440653 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.440605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ca03da-589d-49f1-825e-f5b22caf2b26-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.440854 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.440667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ca03da-589d-49f1-825e-f5b22caf2b26-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.440854 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.440712    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvsvl\" (UniqueName: \"kubernetes.io/projected/20ca03da-589d-49f1-825e-f5b22caf2b26-kube-api-access-vvsvl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.440854 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.440752    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ca03da-589d-49f1-825e-f5b22caf2b26-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.440854 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.440779    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ca03da-589d-49f1-825e-f5b22caf2b26-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.440854 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.440829    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ca03da-589d-49f1-825e-f5b22caf2b26-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.441257 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.441230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ca03da-589d-49f1-825e-f5b22caf2b26-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.441351 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.441254    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ca03da-589d-49f1-825e-f5b22caf2b26-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.441351 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.441309    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ca03da-589d-49f1-825e-f5b22caf2b26-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.443573 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.443549    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ca03da-589d-49f1-825e-f5b22caf2b26-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.443776 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.443758    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ca03da-589d-49f1-825e-f5b22caf2b26-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.448666 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.448642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvsvl\" (UniqueName: \"kubernetes.io/projected/20ca03da-589d-49f1-825e-f5b22caf2b26-kube-api-access-vvsvl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tq5tz\" (UID: \"20ca03da-589d-49f1-825e-f5b22caf2b26\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.540023 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.539929    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:04.814644 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:04.814618    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"]
Sep 15 13:50:04.817130 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:50:04.817099    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ca03da_589d_49f1_825e_f5b22caf2b26.slice/crio-db0e31374f70a7a9129133aed2c427b1e33a34a9308d92beba8d26136f0090a4 WatchSource:0}: Error finding container db0e31374f70a7a9129133aed2c427b1e33a34a9308d92beba8d26136f0090a4: Status 404 returned error can't find the container with id db0e31374f70a7a9129133aed2c427b1e33a34a9308d92beba8d26136f0090a4
Sep 15 13:50:05.482413 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:05.482321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz" event={"ID":"20ca03da-589d-49f1-825e-f5b22caf2b26","Type":"ContainerStarted","Data":"950dd3876b4bacf3aa04e38f68f585017c89384887bc4d64c78ae3f3994744a8"}
Sep 15 13:50:05.482413 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:05.482373    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz" event={"ID":"20ca03da-589d-49f1-825e-f5b22caf2b26","Type":"ContainerStarted","Data":"db0e31374f70a7a9129133aed2c427b1e33a34a9308d92beba8d26136f0090a4"}
Sep 15 13:50:05.499753 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:05.499695    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz" podStartSLOduration=1.216538128 podStartE2EDuration="1.499674212s" totalImagesPullingTime="283.136084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:50:05.498565909 +0000 UTC m=+703.376468729" watchObservedRunningTime="2026-09-15 13:50:05.499674212 +0000 UTC m=+703.377577025"
Sep 15 13:50:05.520784 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:05.520748    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:50:09.223882 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:09.223841    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:50:09.224349 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:09.223900    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:50:09.236263 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:09.236237    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:50:09.518690 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:09.518606    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54"
Sep 15 13:50:11.256848 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:11.256817    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:50:11.358291 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:11.358262    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:50:14.128185 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:14.128133    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:50:14.540695 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:14.540605    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:14.540695 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:14.540670    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:14.553358 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:14.553327    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:50:15.544772 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:50:15.544742    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tq5tz"
Sep 15 13:51:26.346365 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:26.346335    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:27.013881 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:27.013845    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:27.112328 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:27.112287    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:34.067703 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:34.067670    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:34.169442 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:34.169411    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:35.166802 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:35.166764    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:35.265483 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:35.265445    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:50.983333 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:50.983251    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:51.084410 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.084374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:51.185368 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.185320    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:51.353418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.353370    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:51.452971 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.452929    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:51.469702 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.469667    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"]
Sep 15 13:51:51.473227 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.473206    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.476847 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.476822    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4441a1-openshift-default-dockercfg-fzprm\""
Sep 15 13:51:51.493417 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.493386    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"]
Sep 15 13:51:51.548040 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.548003    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-token\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.548040 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.548049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfhzk\" (UniqueName: \"kubernetes.io/projected/22efe0fe-300a-4cdd-81c1-81cc8d379837-kube-api-access-lfhzk\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.548276 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.548070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-data\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.548276 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.548189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-podinfo\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.548276 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.548255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-credential-socket\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.548376 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.548285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-envoy\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.548376 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.548309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22efe0fe-300a-4cdd-81c1-81cc8d379837-istiod-ca-cert\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.548376 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.548347    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-workload-socket\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.548468 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.548382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-workload-certs\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.561450 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.561415    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:51.649656 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.649558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-token\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.649656 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.649599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfhzk\" (UniqueName: \"kubernetes.io/projected/22efe0fe-300a-4cdd-81c1-81cc8d379837-kube-api-access-lfhzk\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.649656 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.649621    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-data\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.649656 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.649644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-podinfo\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.650007 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.649683    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-credential-socket\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.650007 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.649710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-envoy\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.650007 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.649731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22efe0fe-300a-4cdd-81c1-81cc8d379837-istiod-ca-cert\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.650007 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.649756    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-workload-socket\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.650007 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.649781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-workload-certs\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.650292 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.650051    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-data\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.650292 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.650120    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-workload-certs\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.650292 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.650241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-workload-socket\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.650543 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.650522    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-credential-socket\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.650543 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.650530    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:51.650749 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.650728    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22efe0fe-300a-4cdd-81c1-81cc8d379837-istiod-ca-cert\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.652709 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.652681    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-envoy\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.652898 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.652879    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-podinfo\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.657732 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.657713    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-token\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.657870 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.657851    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfhzk\" (UniqueName: \"kubernetes.io/projected/22efe0fe-300a-4cdd-81c1-81cc8d379837-kube-api-access-lfhzk\") pod \"e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") " pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.750491 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.750443    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:51.786255 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.786214    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:51.954384 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:51.954291    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:52.060787 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:52.060689    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"]
Sep 15 13:51:52.063808 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:51:52.063772    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22efe0fe_300a_4cdd_81c1_81cc8d379837.slice/crio-567e02d88221c7bae630b0202bad60f75ff2a12568d0c14ee17bf4d19e8365c1 WatchSource:0}: Error finding container 567e02d88221c7bae630b0202bad60f75ff2a12568d0c14ee17bf4d19e8365c1: Status 404 returned error can't find the container with id 567e02d88221c7bae630b0202bad60f75ff2a12568d0c14ee17bf4d19e8365c1
Sep 15 13:51:52.066529 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:52.066487    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:51:52.066630 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:52.066559    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:51:52.066630 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:52.066590    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:51:52.357370 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:52.357331    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:52.449281 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:52.449245    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:52.769554 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:52.769456    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:52.979593 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:52.979520    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" event={"ID":"22efe0fe-300a-4cdd-81c1-81cc8d379837","Type":"ContainerStarted","Data":"4c95ef6011c45f057e3fcea2e0048349879112e240e87961284ea4d14d8a4a4c"}
Sep 15 13:51:52.979593 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:52.979578    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" event={"ID":"22efe0fe-300a-4cdd-81c1-81cc8d379837","Type":"ContainerStarted","Data":"567e02d88221c7bae630b0202bad60f75ff2a12568d0c14ee17bf4d19e8365c1"}
Sep 15 13:51:53.008091 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:53.008035    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" podStartSLOduration=2.008010283 podStartE2EDuration="2.008010283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:51:53.005631786 +0000 UTC m=+810.883534606" watchObservedRunningTime="2026-09-15 13:51:53.008010283 +0000 UTC m=+810.885913102"
Sep 15 13:51:53.786572 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:53.786539    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:53.791990 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:53.791963    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:53.985290 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:53.985262    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:53.986327 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:53.986308    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:51:54.529402 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:54.529365    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:54.625320 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:54.625285    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:54.718474 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:54.718430    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:54.931989 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:54.931949    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:51:55.024735 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:55.024700    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:55.124838 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:55.124803    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:56.196064 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:56.196030    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:56.297867 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:56.297829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:56.753336 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:56.753306    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:56.841629 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:56.841589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:57.386130 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:57.386078    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:57.445920 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:57.445869    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:57.640683 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:57.640574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:57.742346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:57.742304    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:57.987495 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:57.987391    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:58.092812 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:58.092785    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:58.388329 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:58.388257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:58.575357 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:58.575300    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:58.649573 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:58.649492    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:58.950407 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:58.950319    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:59.085542 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.085512    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:59.156514 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.156478    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:59.204392 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.204311    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t"]
Sep 15 13:51:59.208392 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.208369    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t"
Sep 15 13:51:59.211340 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.211316    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4441a1-dockercfg-4lvwn\""
Sep 15 13:51:59.211459 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.211316    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4441a1\""
Sep 15 13:51:59.220339 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.220313    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t"]
Sep 15 13:51:59.323859 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.323814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfw5n\" (UniqueName: \"kubernetes.io/projected/2f29cf7e-3a8e-4d84-803b-accf8fedca0c-kube-api-access-cfw5n\") pod \"payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t\" (UID: \"2f29cf7e-3a8e-4d84-803b-accf8fedca0c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t"
Sep 15 13:51:59.324072 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.323866    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f29cf7e-3a8e-4d84-803b-accf8fedca0c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t\" (UID: \"2f29cf7e-3a8e-4d84-803b-accf8fedca0c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t"
Sep 15 13:51:59.425456 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.425419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfw5n\" (UniqueName: \"kubernetes.io/projected/2f29cf7e-3a8e-4d84-803b-accf8fedca0c-kube-api-access-cfw5n\") pod \"payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t\" (UID: \"2f29cf7e-3a8e-4d84-803b-accf8fedca0c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t"
Sep 15 13:51:59.425823 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.425466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f29cf7e-3a8e-4d84-803b-accf8fedca0c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t\" (UID: \"2f29cf7e-3a8e-4d84-803b-accf8fedca0c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t"
Sep 15 13:51:59.426339 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.426315    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f29cf7e-3a8e-4d84-803b-accf8fedca0c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t\" (UID: \"2f29cf7e-3a8e-4d84-803b-accf8fedca0c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t"
Sep 15 13:51:59.434237 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.434210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfw5n\" (UniqueName: \"kubernetes.io/projected/2f29cf7e-3a8e-4d84-803b-accf8fedca0c-kube-api-access-cfw5n\") pod \"payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t\" (UID: \"2f29cf7e-3a8e-4d84-803b-accf8fedca0c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t"
Sep 15 13:51:59.518829 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.518744    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t"
Sep 15 13:51:59.699603 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.699573    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:59.792713 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.792635    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t"]
Sep 15 13:51:59.795415 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:51:59.795387    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f29cf7e_3a8e_4d84_803b_accf8fedca0c.slice/crio-ba30bbf7c84ba291881da3b45a36402dbc6e04a7dfb234df6e5136e34ce3ed3c WatchSource:0}: Error finding container ba30bbf7c84ba291881da3b45a36402dbc6e04a7dfb234df6e5136e34ce3ed3c: Status 404 returned error can't find the container with id ba30bbf7c84ba291881da3b45a36402dbc6e04a7dfb234df6e5136e34ce3ed3c
Sep 15 13:51:59.896184 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.896114    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:51:59.999364 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:51:59.999334    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:00.014707 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:00.014678    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t" event={"ID":"2f29cf7e-3a8e-4d84-803b-accf8fedca0c","Type":"ContainerStarted","Data":"ba30bbf7c84ba291881da3b45a36402dbc6e04a7dfb234df6e5136e34ce3ed3c"}
Sep 15 13:52:00.118435 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:00.118406    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:00.431491 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:00.431397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:00.597642 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:00.597593    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:00.807685 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:00.807564    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:01.409789 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:01.409700    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:01.715370 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:01.715273    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:01.814292 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:01.814245    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:02.026702 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:02.026604    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t" event={"ID":"2f29cf7e-3a8e-4d84-803b-accf8fedca0c","Type":"ContainerStarted","Data":"4cd901f435b30ad2eb05ae98aa6d2158835759f7dcfd2a629f5d528a2ceb831d"}
Sep 15 13:52:02.026874 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:02.026724    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t"
Sep 15 13:52:02.046645 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:02.046583    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t" podStartSLOduration=1.7252587240000001 podStartE2EDuration="3.046560696s" totalImagesPullingTime="1.321301972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:52:02.044060652 +0000 UTC m=+819.921963476" watchObservedRunningTime="2026-09-15 13:52:02.046560696 +0000 UTC m=+819.924463517"
Sep 15 13:52:02.434504 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:02.434469    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:03.156851 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:03.156813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:03.258769 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:03.258722    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:03.688286 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:03.688252    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:04.708212 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:04.708173    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:05.711224 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:05.711134    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:05.811845 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:05.811798    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:06.200333 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:06.198956    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:06.297620 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:06.297592    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:06.417610 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:06.417574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:06.806681 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:06.806647    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:07.660934 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.660896    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"]
Sep 15 13:52:07.676382 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.676350    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"]
Sep 15 13:52:07.676692 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.676673    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:07.681217 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.681187    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 13:52:07.685514 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.685492    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 13:52:07.686648 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.685804    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 13:52:07.686947 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.685856    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lkr7h\""
Sep 15 13:52:07.687188 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.685893    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-80a7cc-serving-cert\""
Sep 15 13:52:07.708109 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.707654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:07.777122 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.777075    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:07.809821 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.809785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6354e4e1-2b19-4872-ab2c-4f404a262c05-maas-api-tls\") pod \"maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:07.810276 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.809858    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6354e4e1-2b19-4872-ab2c-4f404a262c05-openshift-service-ca\") pod \"maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:07.810276 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.809975    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6354e4e1-2b19-4872-ab2c-4f404a262c05-metrics-certs\") pod \"maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:07.810276 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.810001    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84gwg\" (UniqueName: \"kubernetes.io/projected/6354e4e1-2b19-4872-ab2c-4f404a262c05-kube-api-access-84gwg\") pod \"maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:07.911047 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.910957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6354e4e1-2b19-4872-ab2c-4f404a262c05-openshift-service-ca\") pod \"maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:07.911203 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.911061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6354e4e1-2b19-4872-ab2c-4f404a262c05-metrics-certs\") pod \"maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:07.911203 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.911081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84gwg\" (UniqueName: \"kubernetes.io/projected/6354e4e1-2b19-4872-ab2c-4f404a262c05-kube-api-access-84gwg\") pod \"maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:07.911203 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.911113    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6354e4e1-2b19-4872-ab2c-4f404a262c05-maas-api-tls\") pod \"maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:07.912130 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.912103    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6354e4e1-2b19-4872-ab2c-4f404a262c05-openshift-service-ca\") pod \"maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:07.913688 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.913658    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6354e4e1-2b19-4872-ab2c-4f404a262c05-maas-api-tls\") pod \"maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:07.913789 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.913692    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6354e4e1-2b19-4872-ab2c-4f404a262c05-metrics-certs\") pod \"maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:07.920151 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.920123    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84gwg\" (UniqueName: \"kubernetes.io/projected/6354e4e1-2b19-4872-ab2c-4f404a262c05-kube-api-access-84gwg\") pod \"maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:07.945926 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:07.945898    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:08.004435 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.004398    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:08.284211 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.280951    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"]
Sep 15 13:52:08.296312 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:52:08.296284    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6354e4e1_2b19_4872_ab2c_4f404a262c05.slice/crio-94483cda49fcc43c58edd24effc6dd93a807601f7a8ecc82576e1ae6a4752179 WatchSource:0}: Error finding container 94483cda49fcc43c58edd24effc6dd93a807601f7a8ecc82576e1ae6a4752179: Status 404 returned error can't find the container with id 94483cda49fcc43c58edd24effc6dd93a807601f7a8ecc82576e1ae6a4752179
Sep 15 13:52:08.348732 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.348701    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:08.447331 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.447264    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:08.538267 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.538186    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"]
Sep 15 13:52:08.542654 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.542635    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"
Sep 15 13:52:08.545376 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.545357    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-80a7cc-dockercfg-w2bjx\""
Sep 15 13:52:08.545591 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.545578    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-80a7cc\""
Sep 15 13:52:08.554409 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.554386    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"]
Sep 15 13:52:08.616508 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.616475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95281f89-c3a2-4518-a7ff-9226aabd196f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5\" (UID: \"95281f89-c3a2-4518-a7ff-9226aabd196f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"
Sep 15 13:52:08.616720 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.616605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhhh6\" (UniqueName: \"kubernetes.io/projected/95281f89-c3a2-4518-a7ff-9226aabd196f-kube-api-access-qhhh6\") pod \"payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5\" (UID: \"95281f89-c3a2-4518-a7ff-9226aabd196f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"
Sep 15 13:52:08.717634 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.717599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95281f89-c3a2-4518-a7ff-9226aabd196f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5\" (UID: \"95281f89-c3a2-4518-a7ff-9226aabd196f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"
Sep 15 13:52:08.717830 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.717785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhhh6\" (UniqueName: \"kubernetes.io/projected/95281f89-c3a2-4518-a7ff-9226aabd196f-kube-api-access-qhhh6\") pod \"payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5\" (UID: \"95281f89-c3a2-4518-a7ff-9226aabd196f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"
Sep 15 13:52:08.718336 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.718310    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95281f89-c3a2-4518-a7ff-9226aabd196f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5\" (UID: \"95281f89-c3a2-4518-a7ff-9226aabd196f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"
Sep 15 13:52:08.728117 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.728092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhhh6\" (UniqueName: \"kubernetes.io/projected/95281f89-c3a2-4518-a7ff-9226aabd196f-kube-api-access-qhhh6\") pod \"payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5\" (UID: \"95281f89-c3a2-4518-a7ff-9226aabd196f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"
Sep 15 13:52:08.818450 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.818366    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:08.855397 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:08.855302    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"
Sep 15 13:52:09.094193 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:09.092830    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg" event={"ID":"6354e4e1-2b19-4872-ab2c-4f404a262c05","Type":"ContainerStarted","Data":"94483cda49fcc43c58edd24effc6dd93a807601f7a8ecc82576e1ae6a4752179"}
Sep 15 13:52:09.185198 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:09.179634    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"]
Sep 15 13:52:09.222525 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:09.222493    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:09.323861 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:09.323794    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:09.638560 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:09.638522    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:09.772854 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:09.768523    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:09.918258 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:09.918211    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:10.010124 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:10.010083    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:10.100712 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:10.100677    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5" event={"ID":"95281f89-c3a2-4518-a7ff-9226aabd196f","Type":"ContainerStarted","Data":"ba798ef3a9ea0a710f856b4d0b4a282bbd9c1137ab60ddd5b4a8ea6480d94936"}
Sep 15 13:52:10.100893 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:10.100734    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5" event={"ID":"95281f89-c3a2-4518-a7ff-9226aabd196f","Type":"ContainerStarted","Data":"455f274829fc56e025a5521a50e523a95a7ee857315964c7001015e17fdf2b63"}
Sep 15 13:52:10.100893 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:10.100775    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"
Sep 15 13:52:10.117851 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:10.117803    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5" podStartSLOduration=2.117786821 podStartE2EDuration="2.117786821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:52:10.115690511 +0000 UTC m=+827.993593342" watchObservedRunningTime="2026-09-15 13:52:10.117786821 +0000 UTC m=+827.995689650"
Sep 15 13:52:10.314835 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:10.314795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:10.700856 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:10.699823    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:10.812022 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:10.811979    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:10.913932 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:10.913891    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:11.053492 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:11.053411    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:11.150487 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:11.150451    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:11.274969 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:11.274927    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:11.618194 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:11.618153    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:12.114382 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:12.114342    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg" event={"ID":"6354e4e1-2b19-4872-ab2c-4f404a262c05","Type":"ContainerStarted","Data":"e676a1c034240b93a4e830ecc3b9e795b7a39ab555350eb8dfc032beaf908d9e"}
Sep 15 13:52:12.114762 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:12.114472    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:12.134650 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:12.134601    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg" podStartSLOduration=2.201002806 podStartE2EDuration="5.134585692s" totalImagesPullingTime="2.933582886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:52:12.132393423 +0000 UTC m=+830.010296254" watchObservedRunningTime="2026-09-15 13:52:12.134585692 +0000 UTC m=+830.012488511"
Sep 15 13:52:12.295877 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:12.295844    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:12.533489 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:12.533404    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:52:13.033619 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:13.033539    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t"
Sep 15 13:52:13.121378 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:13.121298    2582 generic.go:356] "Generic (PLEG): container finished" podID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerID="4c95ef6011c45f057e3fcea2e0048349879112e240e87961284ea4d14d8a4a4c" exitCode=0
Sep 15 13:52:13.121378 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:13.121367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" event={"ID":"22efe0fe-300a-4cdd-81c1-81cc8d379837","Type":"ContainerDied","Data":"4c95ef6011c45f057e3fcea2e0048349879112e240e87961284ea4d14d8a4a4c"}
Sep 15 13:52:13.121837 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:13.121822    2582 scope.go:117] "RemoveContainer" containerID="4c95ef6011c45f057e3fcea2e0048349879112e240e87961284ea4d14d8a4a4c"
Sep 15 13:52:13.122879 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:13.122848    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:52:13.122945 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:13.122915    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:52:13.122980 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:13.122945    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 13:52:13.509123 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:13.509024    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:13.687221 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:13.687184    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:13.781696 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:13.781607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:14.129492 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:14.129460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" event={"ID":"22efe0fe-300a-4cdd-81c1-81cc8d379837","Type":"ContainerStarted","Data":"d0fe4b57b257107510baaa2d556a7cd328dc678e5d16e5ed1a52326e3ad97dc9"}
Sep 15 13:52:14.794694 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:14.794649    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:52:14.799368 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:14.799318    2582 patch_prober.go:28] interesting pod/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 13:52:14.799571 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:14.799432    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 13:52:15.787165 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:15.787103    2582 patch_prober.go:28] interesting pod/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 13:52:15.787646 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:15.787199    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 13:52:16.479633 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:16.479551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:16.581277 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:16.581237    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:16.787488 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:16.787386    2582 patch_prober.go:28] interesting pod/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 13:52:16.787488 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:16.787458    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 13:52:17.112541 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:17.112510    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:17.787001 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:17.786963    2582 patch_prober.go:28] interesting pod/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 13:52:17.787329 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:17.787298    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 13:52:17.900555 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:17.900522    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:18.127664 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:18.127631    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:52:18.771827 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:18.771737    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:18.787047 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:18.787011    2582 patch_prober.go:28] interesting pod/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 13:52:18.787373 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:18.787325    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 13:52:19.465163 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.463637    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"]
Sep 15 13:52:19.468169 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.467611    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:19.472201 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.471640    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2aa332-serving-cert\""
Sep 15 13:52:19.483171 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.483083    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"]
Sep 15 13:52:19.501875 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.501837    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:19.531034 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.530998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc7fb591-a269-4d8e-8260-d109e79a2708-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd\" (UID: \"fc7fb591-a269-4d8e-8260-d109e79a2708\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:19.531269 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.531052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wptcz\" (UniqueName: \"kubernetes.io/projected/fc7fb591-a269-4d8e-8260-d109e79a2708-kube-api-access-wptcz\") pod \"maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd\" (UID: \"fc7fb591-a269-4d8e-8260-d109e79a2708\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:19.531269 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.531084    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7fb591-a269-4d8e-8260-d109e79a2708-metrics-certs\") pod \"maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd\" (UID: \"fc7fb591-a269-4d8e-8260-d109e79a2708\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:19.531269 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.531165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7fb591-a269-4d8e-8260-d109e79a2708-maas-api-tls\") pod \"maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd\" (UID: \"fc7fb591-a269-4d8e-8260-d109e79a2708\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:19.601722 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.601681    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:19.632070 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.632026    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7fb591-a269-4d8e-8260-d109e79a2708-maas-api-tls\") pod \"maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd\" (UID: \"fc7fb591-a269-4d8e-8260-d109e79a2708\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:19.632267 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.632166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc7fb591-a269-4d8e-8260-d109e79a2708-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd\" (UID: \"fc7fb591-a269-4d8e-8260-d109e79a2708\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:19.632267 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:52:19.632217    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-serving-cert: secret "maas-api-e2e-inf-b-2aa332-serving-cert" not found
Sep 15 13:52:19.632267 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.632237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wptcz\" (UniqueName: \"kubernetes.io/projected/fc7fb591-a269-4d8e-8260-d109e79a2708-kube-api-access-wptcz\") pod \"maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd\" (UID: \"fc7fb591-a269-4d8e-8260-d109e79a2708\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:19.632437 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:52:19.632297    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc7fb591-a269-4d8e-8260-d109e79a2708-maas-api-tls podName:fc7fb591-a269-4d8e-8260-d109e79a2708 nodeName:}" failed. No retries permitted until 2026-09-15 13:52:20.132270232 +0000 UTC m=+838.010173048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc7fb591-a269-4d8e-8260-d109e79a2708-maas-api-tls") pod "maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd" (UID: "fc7fb591-a269-4d8e-8260-d109e79a2708") : secret "maas-api-e2e-inf-b-2aa332-serving-cert" not found
Sep 15 13:52:19.632437 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.632319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7fb591-a269-4d8e-8260-d109e79a2708-metrics-certs\") pod \"maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd\" (UID: \"fc7fb591-a269-4d8e-8260-d109e79a2708\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:19.633043 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.633018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc7fb591-a269-4d8e-8260-d109e79a2708-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd\" (UID: \"fc7fb591-a269-4d8e-8260-d109e79a2708\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:19.635064 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.635036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7fb591-a269-4d8e-8260-d109e79a2708-metrics-certs\") pod \"maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd\" (UID: \"fc7fb591-a269-4d8e-8260-d109e79a2708\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:19.641768 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.641726    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wptcz\" (UniqueName: \"kubernetes.io/projected/fc7fb591-a269-4d8e-8260-d109e79a2708-kube-api-access-wptcz\") pod \"maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd\" (UID: \"fc7fb591-a269-4d8e-8260-d109e79a2708\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:19.709559 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.709489    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:19.791311 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.791231    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:52:19.791539 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.791514    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:52:19.792271 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:19.792254    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 13:52:20.075736 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:20.075624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:20.137654 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:20.137616    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7fb591-a269-4d8e-8260-d109e79a2708-maas-api-tls\") pod \"maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd\" (UID: \"fc7fb591-a269-4d8e-8260-d109e79a2708\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:20.140051 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:20.140026    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7fb591-a269-4d8e-8260-d109e79a2708-maas-api-tls\") pod \"maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd\" (UID: \"fc7fb591-a269-4d8e-8260-d109e79a2708\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:20.166640 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:20.166595    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:20.385506 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:20.385468    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:20.492962 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:20.492930    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:20.658590 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:20.658526    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"]
Sep 15 13:52:20.660704 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:52:20.660673    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7fb591_a269_4d8e_8260_d109e79a2708.slice/crio-ef3ccd6b5c9427d86d8234b477d98c226094a80b90c99d5136904aa4696a69ef WatchSource:0}: Error finding container ef3ccd6b5c9427d86d8234b477d98c226094a80b90c99d5136904aa4696a69ef: Status 404 returned error can't find the container with id ef3ccd6b5c9427d86d8234b477d98c226094a80b90c99d5136904aa4696a69ef
Sep 15 13:52:21.109657 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:21.109629    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"
Sep 15 13:52:21.164839 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:21.164762    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd" event={"ID":"fc7fb591-a269-4d8e-8260-d109e79a2708","Type":"ContainerStarted","Data":"ef3ccd6b5c9427d86d8234b477d98c226094a80b90c99d5136904aa4696a69ef"}
Sep 15 13:52:21.326599 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:21.326567    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:21.429410 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:21.429331    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:22.133245 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:22.133209    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:22.172819 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:22.172786    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd" event={"ID":"fc7fb591-a269-4d8e-8260-d109e79a2708","Type":"ContainerStarted","Data":"fee03a406014bdd883be59994498ccf91a1385a92ee5351cc020bdb78d405bc4"}
Sep 15 13:52:22.173021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:22.172893    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:22.191681 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:22.191632    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd" podStartSLOduration=2.647763164 podStartE2EDuration="3.191617181s" totalImagesPullingTime="543.854017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:52:22.190363525 +0000 UTC m=+840.068266349" watchObservedRunningTime="2026-09-15 13:52:22.191617181 +0000 UTC m=+840.069520027"
Sep 15 13:52:23.021840 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:23.021799    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:24.031384 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:24.031345    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:24.715309 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:24.715259    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:25.416628 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:25.416589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:26.200566 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:26.200533    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:26.923655 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:26.923620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:27.672808 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:27.672770    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:28.184276 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:28.184248    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd"
Sep 15 13:52:28.367690 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:28.367656    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:29.064433 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:29.064392    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:29.793525 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:29.793494    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:30.526469 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:30.526432    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:52:33.552838 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:33.552799    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:34.181645 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:34.181610    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:34.265389 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:34.265350    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:36.805672 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:36.805638    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:36.909281 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:36.909250    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:37.494646 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:37.494611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:40.486969 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.486934    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:40.570431 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.570397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:40.575295 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.575264    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"]
Sep 15 13:52:40.579006 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.578986    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:40.581817 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.581788    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a5a300cf-serving-cert\""
Sep 15 13:52:40.589212 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.589188    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"]
Sep 15 13:52:40.636192 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.636126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:40.731525 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.731488    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93358352-3e23-4c51-8a01-a767abbb3f6c-openshift-service-ca\") pod \"maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:40.731525 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.731524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93358352-3e23-4c51-8a01-a767abbb3f6c-metrics-certs\") pod \"maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:40.731740 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.731550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4d6h\" (UniqueName: \"kubernetes.io/projected/93358352-3e23-4c51-8a01-a767abbb3f6c-kube-api-access-x4d6h\") pod \"maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:40.731740 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.731636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93358352-3e23-4c51-8a01-a767abbb3f6c-maas-api-tls\") pod \"maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:40.737448 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.737364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:40.832374 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.832334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93358352-3e23-4c51-8a01-a767abbb3f6c-openshift-service-ca\") pod \"maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:40.832374 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.832375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93358352-3e23-4c51-8a01-a767abbb3f6c-metrics-certs\") pod \"maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:40.832639 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.832457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4d6h\" (UniqueName: \"kubernetes.io/projected/93358352-3e23-4c51-8a01-a767abbb3f6c-kube-api-access-x4d6h\") pod \"maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:40.832639 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.832501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93358352-3e23-4c51-8a01-a767abbb3f6c-maas-api-tls\") pod \"maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:40.833074 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.833050    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93358352-3e23-4c51-8a01-a767abbb3f6c-openshift-service-ca\") pod \"maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:40.834782 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.834762    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93358352-3e23-4c51-8a01-a767abbb3f6c-maas-api-tls\") pod \"maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:40.834909 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.834892    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93358352-3e23-4c51-8a01-a767abbb3f6c-metrics-certs\") pod \"maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:40.840095 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.840074    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4d6h\" (UniqueName: \"kubernetes.io/projected/93358352-3e23-4c51-8a01-a767abbb3f6c-kube-api-access-x4d6h\") pod \"maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:40.893805 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:40.893772    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:41.097280 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:41.097247    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:41.164712 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:41.164683    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"]
Sep 15 13:52:41.166293 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:41.166248    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:41.168123 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:52:41.168096    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93358352_3e23_4c51_8a01_a767abbb3f6c.slice/crio-31daf67dbee1bdf23c3563f9d2650ffb023c80b70a9edf5a415dcf7f84b5c0f5 WatchSource:0}: Error finding container 31daf67dbee1bdf23c3563f9d2650ffb023c80b70a9edf5a415dcf7f84b5c0f5: Status 404 returned error can't find the container with id 31daf67dbee1bdf23c3563f9d2650ffb023c80b70a9edf5a415dcf7f84b5c0f5
Sep 15 13:52:41.269386 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:41.269350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228" event={"ID":"93358352-3e23-4c51-8a01-a767abbb3f6c","Type":"ContainerStarted","Data":"31daf67dbee1bdf23c3563f9d2650ffb023c80b70a9edf5a415dcf7f84b5c0f5"}
Sep 15 13:52:41.521643 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:41.521533    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:42.276755 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:42.276724    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228" event={"ID":"93358352-3e23-4c51-8a01-a767abbb3f6c","Type":"ContainerStarted","Data":"685225cb6ad0828a879038348ea29effcb8b5991df8c88e6f730b25ce81b14cb"}
Sep 15 13:52:42.276925 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:42.276834    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:42.294600 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:42.294557    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228" podStartSLOduration=1.8819954129999998 podStartE2EDuration="2.29454231s" totalImagesPullingTime="412.546897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:52:42.292918838 +0000 UTC m=+860.170821657" watchObservedRunningTime="2026-09-15 13:52:42.29454231 +0000 UTC m=+860.172445129"
Sep 15 13:52:42.450866 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:42.450829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:43.120641 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:43.120611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:44.003613 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:44.003566    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:44.689958 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:44.689919    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:48.288615 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:48.288584    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:52:48.814433 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:48.814393    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:48.857190 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:48.857156    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:52:48.912791 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:48.912753    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:49.025852 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:49.025813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:54.870767 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:54.870732    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:54.986495 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:54.986461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:55.071752 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:55.071716    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:55.194459 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:55.194352    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:52:57.135898 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:52:57.135861    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:00.951759 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:00.951721    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:01.051581 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:01.051535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:02.693627 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:02.693601    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:03.587582 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:03.587549    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:04.264535 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:04.264497    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:04.524785 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:04.524704    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:53:04.953649 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:04.953612    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:05.844677 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:05.844638    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:06.491508 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:06.491479    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:07.766561 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:07.766523    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:09.893553 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:09.893517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:09.994756 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:09.994724    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:10.314385 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.314296    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"]
Sep 15 13:53:10.314549 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.314506    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228" podUID="93358352-3e23-4c51-8a01-a767abbb3f6c" containerName="maas-api" containerID="cri-o://685225cb6ad0828a879038348ea29effcb8b5991df8c88e6f730b25ce81b14cb" gracePeriod=30
Sep 15 13:53:10.563241 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.563120    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:53:10.608306 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.608273    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93358352-3e23-4c51-8a01-a767abbb3f6c-metrics-certs\") pod \"93358352-3e23-4c51-8a01-a767abbb3f6c\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") "
Sep 15 13:53:10.608495 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.608342    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93358352-3e23-4c51-8a01-a767abbb3f6c-maas-api-tls\") pod \"93358352-3e23-4c51-8a01-a767abbb3f6c\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") "
Sep 15 13:53:10.608495 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.608420    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93358352-3e23-4c51-8a01-a767abbb3f6c-openshift-service-ca\") pod \"93358352-3e23-4c51-8a01-a767abbb3f6c\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") "
Sep 15 13:53:10.608495 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.608452    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4d6h\" (UniqueName: \"kubernetes.io/projected/93358352-3e23-4c51-8a01-a767abbb3f6c-kube-api-access-x4d6h\") pod \"93358352-3e23-4c51-8a01-a767abbb3f6c\" (UID: \"93358352-3e23-4c51-8a01-a767abbb3f6c\") "
Sep 15 13:53:10.608861 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.608831    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93358352-3e23-4c51-8a01-a767abbb3f6c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93358352-3e23-4c51-8a01-a767abbb3f6c" (UID: "93358352-3e23-4c51-8a01-a767abbb3f6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 13:53:10.610469 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.610442    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93358352-3e23-4c51-8a01-a767abbb3f6c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "93358352-3e23-4c51-8a01-a767abbb3f6c" (UID: "93358352-3e23-4c51-8a01-a767abbb3f6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:53:10.610658 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.610636    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93358352-3e23-4c51-8a01-a767abbb3f6c-kube-api-access-x4d6h" (OuterVolumeSpecName: "kube-api-access-x4d6h") pod "93358352-3e23-4c51-8a01-a767abbb3f6c" (UID: "93358352-3e23-4c51-8a01-a767abbb3f6c"). InnerVolumeSpecName "kube-api-access-x4d6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:53:10.610800 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.610785    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93358352-3e23-4c51-8a01-a767abbb3f6c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93358352-3e23-4c51-8a01-a767abbb3f6c" (UID: "93358352-3e23-4c51-8a01-a767abbb3f6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:53:10.709339 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.709298    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93358352-3e23-4c51-8a01-a767abbb3f6c-openshift-service-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:53:10.709339 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.709332    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4d6h\" (UniqueName: \"kubernetes.io/projected/93358352-3e23-4c51-8a01-a767abbb3f6c-kube-api-access-x4d6h\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:53:10.709339 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.709343    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93358352-3e23-4c51-8a01-a767abbb3f6c-metrics-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:53:10.709563 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:10.709352    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93358352-3e23-4c51-8a01-a767abbb3f6c-maas-api-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:53:11.435118 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:11.435087    2582 generic.go:356] "Generic (PLEG): container finished" podID="93358352-3e23-4c51-8a01-a767abbb3f6c" containerID="685225cb6ad0828a879038348ea29effcb8b5991df8c88e6f730b25ce81b14cb" exitCode=0
Sep 15 13:53:11.435545 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:11.435132    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"
Sep 15 13:53:11.435545 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:11.435206    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228" event={"ID":"93358352-3e23-4c51-8a01-a767abbb3f6c","Type":"ContainerDied","Data":"685225cb6ad0828a879038348ea29effcb8b5991df8c88e6f730b25ce81b14cb"}
Sep 15 13:53:11.435545 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:11.435260    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228" event={"ID":"93358352-3e23-4c51-8a01-a767abbb3f6c","Type":"ContainerDied","Data":"31daf67dbee1bdf23c3563f9d2650ffb023c80b70a9edf5a415dcf7f84b5c0f5"}
Sep 15 13:53:11.435545 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:11.435289    2582 scope.go:117] "RemoveContainer" containerID="685225cb6ad0828a879038348ea29effcb8b5991df8c88e6f730b25ce81b14cb"
Sep 15 13:53:11.445513 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:11.445491    2582 scope.go:117] "RemoveContainer" containerID="685225cb6ad0828a879038348ea29effcb8b5991df8c88e6f730b25ce81b14cb"
Sep 15 13:53:11.445838 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:53:11.445808    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685225cb6ad0828a879038348ea29effcb8b5991df8c88e6f730b25ce81b14cb\": container with ID starting with 685225cb6ad0828a879038348ea29effcb8b5991df8c88e6f730b25ce81b14cb not found: ID does not exist" containerID="685225cb6ad0828a879038348ea29effcb8b5991df8c88e6f730b25ce81b14cb"
Sep 15 13:53:11.445943 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:11.445851    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685225cb6ad0828a879038348ea29effcb8b5991df8c88e6f730b25ce81b14cb"} err="failed to get container status \"685225cb6ad0828a879038348ea29effcb8b5991df8c88e6f730b25ce81b14cb\": rpc error: code = NotFound desc = could not find container \"685225cb6ad0828a879038348ea29effcb8b5991df8c88e6f730b25ce81b14cb\": container with ID starting with 685225cb6ad0828a879038348ea29effcb8b5991df8c88e6f730b25ce81b14cb not found: ID does not exist"
Sep 15 13:53:11.455865 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:11.455840    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"]
Sep 15 13:53:11.460601 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:11.460574    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a5a300cf-5cbb546686-l9228"]
Sep 15 13:53:12.254018 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:12.253985    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:12.354567 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:12.354536    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:12.642489 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:12.642458    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93358352-3e23-4c51-8a01-a767abbb3f6c" path="/var/lib/kubelet/pods/93358352-3e23-4c51-8a01-a767abbb3f6c/volumes"
Sep 15 13:53:12.759315 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:12.759282    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:13.492560 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:13.492525    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:13.616432 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:13.616398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:13.717392 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:13.717329    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:14.416028 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.415989    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"]
Sep 15 13:53:14.416420 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.416406    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93358352-3e23-4c51-8a01-a767abbb3f6c" containerName="maas-api"
Sep 15 13:53:14.416473 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.416422    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="93358352-3e23-4c51-8a01-a767abbb3f6c" containerName="maas-api"
Sep 15 13:53:14.416515 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.416491    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="93358352-3e23-4c51-8a01-a767abbb3f6c" containerName="maas-api"
Sep 15 13:53:14.419484 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.419467    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.421847 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.421820    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1293e7\"/\"simulator-w0-1293e7-kserve-self-signed-certs\""
Sep 15 13:53:14.422647 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.422627    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1293e7\"/\"default-dockercfg-m9vk5\""
Sep 15 13:53:14.422748 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.422668    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1293e7\"/\"kube-root-ca.crt\""
Sep 15 13:53:14.422748 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.422671    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1293e7\"/\"openshift-service-ca.crt\""
Sep 15 13:53:14.429392 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.429368    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"]
Sep 15 13:53:14.434158 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.434099    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:14.539257 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.539224    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:14.544802 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.544768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-dshm\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.544938 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.544804    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-model-cache\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.544938 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.544869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-tls-certs\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.544938 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.544920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-tmp-dir\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.544938 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.544936    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t4kd\" (UniqueName: \"kubernetes.io/projected/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-kube-api-access-2t4kd\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.545081 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.544966    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-home\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.645418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.645385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-dshm\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.645418 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.645421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-model-cache\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.645661 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.645469    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-tls-certs\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.645661 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.645508    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-tmp-dir\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.645661 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.645530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t4kd\" (UniqueName: \"kubernetes.io/projected/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-kube-api-access-2t4kd\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.645661 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.645558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-home\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.645888 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.645866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-model-cache\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.645951 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.645926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-tmp-dir\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.646006 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.645963    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-home\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.647663 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.647642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-dshm\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.647994 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.647975    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-tls-certs\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.653414 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.653397    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t4kd\" (UniqueName: \"kubernetes.io/projected/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-kube-api-access-2t4kd\") pod \"simulator-w0-1293e7-kserve-5746489596-42tbl\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") " pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:14.729498 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:14.729385    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:15.007679 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.007650    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"]
Sep 15 13:53:15.010549 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:53:15.010511    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fb9331e_e91a_44cc_abe6_7fec8bef0b71.slice/crio-507b061568fab146634cdc000d7c217cb4c7d999eef456935bc4f6abafcac935 WatchSource:0}: Error finding container 507b061568fab146634cdc000d7c217cb4c7d999eef456935bc4f6abafcac935: Status 404 returned error can't find the container with id 507b061568fab146634cdc000d7c217cb4c7d999eef456935bc4f6abafcac935
Sep 15 13:53:15.173308 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.173272    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:15.458063 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.458031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl" event={"ID":"7fb9331e-e91a-44cc-abe6-7fec8bef0b71","Type":"ContainerStarted","Data":"0bf084195c0ca6a6afbbff93577810e72b29695e6034acd8fb999cca14c32956"}
Sep 15 13:53:15.458063 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.458070    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl" event={"ID":"7fb9331e-e91a-44cc-abe6-7fec8bef0b71","Type":"ContainerStarted","Data":"507b061568fab146634cdc000d7c217cb4c7d999eef456935bc4f6abafcac935"}
Sep 15 13:53:15.477661 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.477599    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl" podStartSLOduration=1.477580076 podStartE2EDuration="1.477580076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:53:15.47592279 +0000 UTC m=+893.353825610" watchObservedRunningTime="2026-09-15 13:53:15.477580076 +0000 UTC m=+893.355482928"
Sep 15 13:53:15.869016 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.868977    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"]
Sep 15 13:53:15.871445 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.871429    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:15.874011 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.873987    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-707dcc5c-serving-cert\""
Sep 15 13:53:15.881816 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.881790    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"]
Sep 15 13:53:15.959795 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.959764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5h84\" (UniqueName: \"kubernetes.io/projected/8da2e6fb-bd6a-4750-8779-3fd5512ee121-kube-api-access-l5h84\") pod \"maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:15.959982 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.959822    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-metrics-certs\") pod \"maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:15.959982 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.959882    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da2e6fb-bd6a-4750-8779-3fd5512ee121-openshift-service-ca\") pod \"maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:15.959982 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.959921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-maas-api-tls\") pod \"maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:15.970680 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:15.970646    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:16.061033 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.060989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-maas-api-tls\") pod \"maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:16.061220 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.061090    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5h84\" (UniqueName: \"kubernetes.io/projected/8da2e6fb-bd6a-4750-8779-3fd5512ee121-kube-api-access-l5h84\") pod \"maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:16.061220 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.061162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-metrics-certs\") pod \"maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:16.061220 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.061211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da2e6fb-bd6a-4750-8779-3fd5512ee121-openshift-service-ca\") pod \"maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:16.061421 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:53:16.061160    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-serving-cert: secret "maas-api-e2e-ait-707dcc5c-serving-cert" not found
Sep 15 13:53:16.061421 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:53:16.061308    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-maas-api-tls podName:8da2e6fb-bd6a-4750-8779-3fd5512ee121 nodeName:}" failed. No retries permitted until 2026-09-15 13:53:16.561286307 +0000 UTC m=+894.439189123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-maas-api-tls") pod "maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2" (UID: "8da2e6fb-bd6a-4750-8779-3fd5512ee121") : secret "maas-api-e2e-ait-707dcc5c-serving-cert" not found
Sep 15 13:53:16.062042 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.062017    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da2e6fb-bd6a-4750-8779-3fd5512ee121-openshift-service-ca\") pod \"maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:16.064220 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.064188    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-metrics-certs\") pod \"maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:16.070560 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.070530    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:16.071022 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.071000    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5h84\" (UniqueName: \"kubernetes.io/projected/8da2e6fb-bd6a-4750-8779-3fd5512ee121-kube-api-access-l5h84\") pod \"maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:16.170812 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.170726    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:16.425008 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.424919    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:16.524783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.524753    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:16.567361 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.567318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-maas-api-tls\") pod \"maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:16.569783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.569756    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-maas-api-tls\") pod \"maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:16.783354 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:16.783278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:17.069097 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:17.069056    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"]
Sep 15 13:53:17.070918 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:53:17.070878    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da2e6fb_bd6a_4750_8779_3fd5512ee121.slice/crio-59533784db78322b08619c712422cbe0a41d8df108fdc3c2aea8090d358ee30d WatchSource:0}: Error finding container 59533784db78322b08619c712422cbe0a41d8df108fdc3c2aea8090d358ee30d: Status 404 returned error can't find the container with id 59533784db78322b08619c712422cbe0a41d8df108fdc3c2aea8090d358ee30d
Sep 15 13:53:17.469973 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:17.469931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2" event={"ID":"8da2e6fb-bd6a-4750-8779-3fd5512ee121","Type":"ContainerStarted","Data":"59533784db78322b08619c712422cbe0a41d8df108fdc3c2aea8090d358ee30d"}
Sep 15 13:53:18.333051 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:18.333017    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:53:18.477563 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:18.477527    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2" event={"ID":"8da2e6fb-bd6a-4750-8779-3fd5512ee121","Type":"ContainerStarted","Data":"e2f164267e618934d8b2415459bde8e0c5b748570d7e870c6b184e1967f98c1a"}
Sep 15 13:53:18.477739 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:18.477628    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:18.500070 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:18.500005    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2" podStartSLOduration=2.786267198 podStartE2EDuration="3.49998756s" totalImagesPullingTime="713.720362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:53:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:53:18.499304135 +0000 UTC m=+896.377206957" watchObservedRunningTime="2026-09-15 13:53:18.49998756 +0000 UTC m=+896.377890380"
Sep 15 13:53:18.692441 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:18.692403    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:18.797541 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:18.797508    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:19.058439 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:19.058340    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:24.488481 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:24.488451    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:24.730379 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:24.730341    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:24.730379 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:24.730389    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:24.743101 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:24.743024    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:25.526365 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:25.526334    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 13:53:26.922905 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:26.922870    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"]
Sep 15 13:53:26.927597 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:26.927576    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:26.931082 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:26.931055    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-66261e\"/\"simulator-w4-66261e-kserve-self-signed-certs\""
Sep 15 13:53:26.931239 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:26.931084    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-66261e\"/\"default-dockercfg-tjnkz\""
Sep 15 13:53:26.931528 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:26.931507    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-66261e\"/\"kube-root-ca.crt\""
Sep 15 13:53:26.931598 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:26.931531    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-66261e\"/\"openshift-service-ca.crt\""
Sep 15 13:53:26.937002 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:26.936976    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"]
Sep 15 13:53:26.957460 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:26.957435    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:27.063624 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.063587    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-dshm\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.063821 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.063707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-tmp-dir\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.063821 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.063735    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-home\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.063821 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.063765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdmw6\" (UniqueName: \"kubernetes.io/projected/cac3cc33-9acb-4215-8db9-3db1bc0be41f-kube-api-access-zdmw6\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.063821 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.063798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-model-cache\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.064048 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.063907    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac3cc33-9acb-4215-8db9-3db1bc0be41f-tls-certs\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.165017 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.164975    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-tmp-dir\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.165244 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.165025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-home\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.165244 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.165190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdmw6\" (UniqueName: \"kubernetes.io/projected/cac3cc33-9acb-4215-8db9-3db1bc0be41f-kube-api-access-zdmw6\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.165386 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.165258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-model-cache\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.165386 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.165347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac3cc33-9acb-4215-8db9-3db1bc0be41f-tls-certs\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.165519 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.165389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-dshm\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.165519 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.165447    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-home\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.165519 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.165494    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-tmp-dir\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.165827 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.165802    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-model-cache\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.167579 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.167548    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-dshm\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.168366 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.168346    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac3cc33-9acb-4215-8db9-3db1bc0be41f-tls-certs\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.173264 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.173207    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdmw6\" (UniqueName: \"kubernetes.io/projected/cac3cc33-9acb-4215-8db9-3db1bc0be41f-kube-api-access-zdmw6\") pod \"simulator-w4-66261e-kserve-6f9cbb5679-tw72q\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") " pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.238731 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.238697    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:27.514348 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.514323    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"]
Sep 15 13:53:27.516431 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:53:27.516402    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcac3cc33_9acb_4215_8db9_3db1bc0be41f.slice/crio-14396295f38b59c9e7316418175decbd5225c984f17d3aa9e81f33e201262ca3 WatchSource:0}: Error finding container 14396295f38b59c9e7316418175decbd5225c984f17d3aa9e81f33e201262ca3: Status 404 returned error can't find the container with id 14396295f38b59c9e7316418175decbd5225c984f17d3aa9e81f33e201262ca3
Sep 15 13:53:27.526453 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:27.526428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q" event={"ID":"cac3cc33-9acb-4215-8db9-3db1bc0be41f","Type":"ContainerStarted","Data":"14396295f38b59c9e7316418175decbd5225c984f17d3aa9e81f33e201262ca3"}
Sep 15 13:53:28.533091 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:28.533056    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q" event={"ID":"cac3cc33-9acb-4215-8db9-3db1bc0be41f","Type":"ContainerStarted","Data":"a0ade85d7d3b5b2ca85cef044bc37fd737c05e976f803b22cdc8cab91d0b84bf"}
Sep 15 13:53:28.551748 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:28.551702    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q" podStartSLOduration=2.551688266 podStartE2EDuration="2.551688266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:53:28.55089266 +0000 UTC m=+906.428795478" watchObservedRunningTime="2026-09-15 13:53:28.551688266 +0000 UTC m=+906.429591084"
Sep 15 13:53:34.633865 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:34.633826    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:53:37.239217 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:37.239178    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:37.239601 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:37.239242    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:37.251782 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:37.251754    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:37.597860 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:37.597827    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 13:53:37.891914 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:37.891835    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:38.702969 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:38.702931    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:39.422495 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:39.422454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:40.194916 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:40.194883    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:40.992368 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:40.992332    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:41.707290 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:41.707253    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:42.592544 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:42.592510    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:43.425060 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:43.425026    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:44.102405 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:44.102375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:44.776996 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:44.776959    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:45.329936 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.329904    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"]
Sep 15 13:53:45.330193 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.330166    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2" podUID="8da2e6fb-bd6a-4750-8779-3fd5512ee121" containerName="maas-api" containerID="cri-o://e2f164267e618934d8b2415459bde8e0c5b748570d7e870c6b184e1967f98c1a" gracePeriod=30
Sep 15 13:53:45.357686 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.357654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:45.432648 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.432617    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:45.532430 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.532387    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:45.585735 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.585715    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:45.588355 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.588327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:45.627906 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.627868    2582 generic.go:356] "Generic (PLEG): container finished" podID="8da2e6fb-bd6a-4750-8779-3fd5512ee121" containerID="e2f164267e618934d8b2415459bde8e0c5b748570d7e870c6b184e1967f98c1a" exitCode=0
Sep 15 13:53:45.628084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.627932    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"
Sep 15 13:53:45.628084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.627956    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2" event={"ID":"8da2e6fb-bd6a-4750-8779-3fd5512ee121","Type":"ContainerDied","Data":"e2f164267e618934d8b2415459bde8e0c5b748570d7e870c6b184e1967f98c1a"}
Sep 15 13:53:45.628084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.628009    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2" event={"ID":"8da2e6fb-bd6a-4750-8779-3fd5512ee121","Type":"ContainerDied","Data":"59533784db78322b08619c712422cbe0a41d8df108fdc3c2aea8090d358ee30d"}
Sep 15 13:53:45.628084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.628043    2582 scope.go:117] "RemoveContainer" containerID="e2f164267e618934d8b2415459bde8e0c5b748570d7e870c6b184e1967f98c1a"
Sep 15 13:53:45.632491 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.632470    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-maas-api-tls\") pod \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") "
Sep 15 13:53:45.632608 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.632514    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-metrics-certs\") pod \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") "
Sep 15 13:53:45.632608 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.632533    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da2e6fb-bd6a-4750-8779-3fd5512ee121-openshift-service-ca\") pod \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") "
Sep 15 13:53:45.632727 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.632608    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5h84\" (UniqueName: \"kubernetes.io/projected/8da2e6fb-bd6a-4750-8779-3fd5512ee121-kube-api-access-l5h84\") pod \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\" (UID: \"8da2e6fb-bd6a-4750-8779-3fd5512ee121\") "
Sep 15 13:53:45.633020 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.632988    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da2e6fb-bd6a-4750-8779-3fd5512ee121-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8da2e6fb-bd6a-4750-8779-3fd5512ee121" (UID: "8da2e6fb-bd6a-4750-8779-3fd5512ee121"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 13:53:45.634665 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.634641    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8da2e6fb-bd6a-4750-8779-3fd5512ee121" (UID: "8da2e6fb-bd6a-4750-8779-3fd5512ee121"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:53:45.634992 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.634973    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8da2e6fb-bd6a-4750-8779-3fd5512ee121" (UID: "8da2e6fb-bd6a-4750-8779-3fd5512ee121"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:53:45.635059 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.635022    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da2e6fb-bd6a-4750-8779-3fd5512ee121-kube-api-access-l5h84" (OuterVolumeSpecName: "kube-api-access-l5h84") pod "8da2e6fb-bd6a-4750-8779-3fd5512ee121" (UID: "8da2e6fb-bd6a-4750-8779-3fd5512ee121"). InnerVolumeSpecName "kube-api-access-l5h84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:53:45.638587 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.638566    2582 scope.go:117] "RemoveContainer" containerID="e2f164267e618934d8b2415459bde8e0c5b748570d7e870c6b184e1967f98c1a"
Sep 15 13:53:45.638887 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:53:45.638861    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f164267e618934d8b2415459bde8e0c5b748570d7e870c6b184e1967f98c1a\": container with ID starting with e2f164267e618934d8b2415459bde8e0c5b748570d7e870c6b184e1967f98c1a not found: ID does not exist" containerID="e2f164267e618934d8b2415459bde8e0c5b748570d7e870c6b184e1967f98c1a"
Sep 15 13:53:45.638947 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.638894    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f164267e618934d8b2415459bde8e0c5b748570d7e870c6b184e1967f98c1a"} err="failed to get container status \"e2f164267e618934d8b2415459bde8e0c5b748570d7e870c6b184e1967f98c1a\": rpc error: code = NotFound desc = could not find container \"e2f164267e618934d8b2415459bde8e0c5b748570d7e870c6b184e1967f98c1a\": container with ID starting with e2f164267e618934d8b2415459bde8e0c5b748570d7e870c6b184e1967f98c1a not found: ID does not exist"
Sep 15 13:53:45.734190 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.734156    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-maas-api-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:53:45.734190 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.734186    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da2e6fb-bd6a-4750-8779-3fd5512ee121-metrics-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:53:45.734190 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.734195    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da2e6fb-bd6a-4750-8779-3fd5512ee121-openshift-service-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:53:45.734422 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.734207    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5h84\" (UniqueName: \"kubernetes.io/projected/8da2e6fb-bd6a-4750-8779-3fd5512ee121-kube-api-access-l5h84\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:53:45.960170 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.960119    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"]
Sep 15 13:53:45.969580 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:45.969545    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-707dcc5c-765f5f8544-mjzg2"]
Sep 15 13:53:46.037261 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:46.037229    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:46.643103 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:46.643069    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8da2e6fb-bd6a-4750-8779-3fd5512ee121" path="/var/lib/kubelet/pods/8da2e6fb-bd6a-4750-8779-3fd5512ee121/volumes"
Sep 15 13:53:46.698330 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:46.698295    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:46.799236 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:46.799203    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:47.136576 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:47.136544    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:47.433507 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:47.433405    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:47.530486 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:47.530459    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:47.718007 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:47.717926    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:48.599650 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:48.599620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:49.482308 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:49.482268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:49.899534 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:49.899497    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:49.991550 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:49.991510    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:50.309936 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:50.309839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:51.095302 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.095260    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"]
Sep 15 13:53:51.095879 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.095857    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da2e6fb-bd6a-4750-8779-3fd5512ee121" containerName="maas-api"
Sep 15 13:53:51.095944 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.095882    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da2e6fb-bd6a-4750-8779-3fd5512ee121" containerName="maas-api"
Sep 15 13:53:51.096032 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.096011    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8da2e6fb-bd6a-4750-8779-3fd5512ee121" containerName="maas-api"
Sep 15 13:53:51.099479 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.099451    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.101734 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.101712    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-974680ec-serving-cert\""
Sep 15 13:53:51.112831 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.112795    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"]
Sep 15 13:53:51.134833 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.134798    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:51.185943 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.185905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df48bd0f-2591-49a5-9dab-694f8df66af5-openshift-service-ca\") pod \"maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.186100 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.185956    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-metrics-certs\") pod \"maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.186100 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.186021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-maas-api-tls\") pod \"maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.186204 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.186095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttts5\" (UniqueName: \"kubernetes.io/projected/df48bd0f-2591-49a5-9dab-694f8df66af5-kube-api-access-ttts5\") pod \"maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.235499 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.235461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:51.286935 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.286891    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-maas-api-tls\") pod \"maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.287102 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.286972    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttts5\" (UniqueName: \"kubernetes.io/projected/df48bd0f-2591-49a5-9dab-694f8df66af5-kube-api-access-ttts5\") pod \"maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.287102 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.287040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df48bd0f-2591-49a5-9dab-694f8df66af5-openshift-service-ca\") pod \"maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.287102 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:53:51.287072    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-serving-cert: secret "maas-api-e2e-derive-974680ec-serving-cert" not found
Sep 15 13:53:51.287276 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.287103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-metrics-certs\") pod \"maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.287276 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:53:51.287176    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-maas-api-tls podName:df48bd0f-2591-49a5-9dab-694f8df66af5 nodeName:}" failed. No retries permitted until 2026-09-15 13:53:51.787154114 +0000 UTC m=+929.665056927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-maas-api-tls") pod "maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s" (UID: "df48bd0f-2591-49a5-9dab-694f8df66af5") : secret "maas-api-e2e-derive-974680ec-serving-cert" not found
Sep 15 13:53:51.287817 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.287793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df48bd0f-2591-49a5-9dab-694f8df66af5-openshift-service-ca\") pod \"maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.289677 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.289659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-metrics-certs\") pod \"maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.295488 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.295466    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttts5\" (UniqueName: \"kubernetes.io/projected/df48bd0f-2591-49a5-9dab-694f8df66af5-kube-api-access-ttts5\") pod \"maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.328957 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.328928    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:53:51.632133 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.632101    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:51.746891 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.746855    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:51.792642 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.792544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-maas-api-tls\") pod \"maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.794870 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.794842    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-maas-api-tls\") pod \"maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:51.842560 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:51.842529    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:52.011668 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:52.011584    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:52.282829 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:52.282802    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"]
Sep 15 13:53:52.285225 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:53:52.285191    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf48bd0f_2591_49a5_9dab_694f8df66af5.slice/crio-2a15d5194293cfa960a1476ff7bbec3c0997587daa529b04067606e92a728163 WatchSource:0}: Error finding container 2a15d5194293cfa960a1476ff7bbec3c0997587daa529b04067606e92a728163: Status 404 returned error can't find the container with id 2a15d5194293cfa960a1476ff7bbec3c0997587daa529b04067606e92a728163
Sep 15 13:53:52.300540 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:52.300511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:52.672591 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:52.672551    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s" event={"ID":"df48bd0f-2591-49a5-9dab-694f8df66af5","Type":"ContainerStarted","Data":"2a15d5194293cfa960a1476ff7bbec3c0997587daa529b04067606e92a728163"}
Sep 15 13:53:52.992363 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:52.992275    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:53.679056 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:53.679019    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s" event={"ID":"df48bd0f-2591-49a5-9dab-694f8df66af5","Type":"ContainerStarted","Data":"e82a2a89405c73a67de9fdcd483fe9704deb98a6b678e13ecdc2028aad8486a6"}
Sep 15 13:53:53.679457 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:53.679163    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:53:53.696150 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:53.696095    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s" podStartSLOduration=2.309406848 podStartE2EDuration="2.696077561s" totalImagesPullingTime="386.670713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:53:53.695014232 +0000 UTC m=+931.572917053" watchObservedRunningTime="2026-09-15 13:53:53.696077561 +0000 UTC m=+931.573980381"
Sep 15 13:53:53.715200 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:53.711260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:54.453784 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:54.453746    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:54.554962 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:54.554923    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:56.385575 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:56.385538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:56.444493 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:56.444455    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:57.908950 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:57.908908    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"]
Sep 15 13:53:57.912801 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:57.912774    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:57.916063 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:57.916032    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-80a7cc\"/\"default-dockercfg-mkpx6\""
Sep 15 13:53:57.916209 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:57.916068    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-80a7cc\"/\"openshift-service-ca.crt\""
Sep 15 13:53:57.916209 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:57.916121    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-80a7cc\"/\"autores-model-80a7cc-kserve-self-signed-certs\""
Sep 15 13:53:57.916311 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:57.916204    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-80a7cc\"/\"kube-root-ca.crt\""
Sep 15 13:53:57.924450 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:57.924426    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"]
Sep 15 13:53:58.052867 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.052830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgf2t\" (UniqueName: \"kubernetes.io/projected/eed098ef-4421-4022-8876-a148541b2395-kube-api-access-kgf2t\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.053084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.052888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-dshm\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.053084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.052916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-home\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.053084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.052962    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-model-cache\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.053084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.053031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed098ef-4421-4022-8876-a148541b2395-tls-certs\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.053084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.053067    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-tmp-dir\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.123563 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.123528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:53:58.153923 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.153889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgf2t\" (UniqueName: \"kubernetes.io/projected/eed098ef-4421-4022-8876-a148541b2395-kube-api-access-kgf2t\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.154085 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.153932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-dshm\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.154085 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.153953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-home\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.154085 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.153996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-model-cache\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.154085 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.154014    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed098ef-4421-4022-8876-a148541b2395-tls-certs\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.154085 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.154047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-tmp-dir\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.154448 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.154424    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-home\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.154518 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.154446    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-tmp-dir\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.154518 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.154506    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-model-cache\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.156237 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.156214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-dshm\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.156505 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.156487    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed098ef-4421-4022-8876-a148541b2395-tls-certs\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.162382 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.162335    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgf2t\" (UniqueName: \"kubernetes.io/projected/eed098ef-4421-4022-8876-a148541b2395-kube-api-access-kgf2t\") pod \"autores-model-80a7cc-kserve-5446f869c5-hmrqh\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") " pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.221743 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.221703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:53:58.495783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.495758    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"]
Sep 15 13:53:58.497729 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:53:58.497694    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeed098ef_4421_4022_8876_a148541b2395.slice/crio-309cba632ae6a33a20251713d31f2fa81eebb85103867ea6a456a58cd5708518 WatchSource:0}: Error finding container 309cba632ae6a33a20251713d31f2fa81eebb85103867ea6a456a58cd5708518: Status 404 returned error can't find the container with id 309cba632ae6a33a20251713d31f2fa81eebb85103867ea6a456a58cd5708518
Sep 15 13:53:58.707357 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.707321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh" event={"ID":"eed098ef-4421-4022-8876-a148541b2395","Type":"ContainerStarted","Data":"5ae650b678154bee204884c107d40d956ad935ecd278a25bacb165b093204b2c"}
Sep 15 13:53:58.707565 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.707367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh" event={"ID":"eed098ef-4421-4022-8876-a148541b2395","Type":"ContainerStarted","Data":"309cba632ae6a33a20251713d31f2fa81eebb85103867ea6a456a58cd5708518"}
Sep 15 13:53:58.728673 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:58.728619    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh" podStartSLOduration=1.7286027069999998 podStartE2EDuration="1.728602707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:53:58.727985468 +0000 UTC m=+936.605888287" watchObservedRunningTime="2026-09-15 13:53:58.728602707 +0000 UTC m=+936.606505522"
Sep 15 13:53:59.690699 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:53:59.690669    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:54:05.962230 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:05.962192    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:06.060633 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:06.060600    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:08.221936 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:08.221899    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:54:08.221936 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:08.221947    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:54:08.234393 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:08.234366    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:54:08.773705 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:08.773673    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:54:10.695595 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:10.695552    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:12.157389 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:12.157357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:13.551909 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:13.551872    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:14.393584 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:14.393551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:15.190837 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:15.190801    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:16.003821 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:16.003783    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:16.680659 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:16.680622    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:16.931096 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:16.931010    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:54:17.386709 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:17.386675    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:18.198363 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:18.198332    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:18.900329 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:18.900289    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:19.594483 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:19.594450    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:21.311785 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:21.311754    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:21.881608 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:21.881572    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"]
Sep 15 13:54:21.881843 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:21.881816    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s" podUID="df48bd0f-2591-49a5-9dab-694f8df66af5" containerName="maas-api" containerID="cri-o://e82a2a89405c73a67de9fdcd483fe9704deb98a6b678e13ecdc2028aad8486a6" gracePeriod=30
Sep 15 13:54:21.914116 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:21.914083    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:22.006396 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.006332    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:22.095329 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.095296    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:22.139771 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.139706    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:54:22.189944 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.189913    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-maas-api-tls\") pod \"df48bd0f-2591-49a5-9dab-694f8df66af5\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") "
Sep 15 13:54:22.190114 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.189958    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df48bd0f-2591-49a5-9dab-694f8df66af5-openshift-service-ca\") pod \"df48bd0f-2591-49a5-9dab-694f8df66af5\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") "
Sep 15 13:54:22.190114 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.189993    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttts5\" (UniqueName: \"kubernetes.io/projected/df48bd0f-2591-49a5-9dab-694f8df66af5-kube-api-access-ttts5\") pod \"df48bd0f-2591-49a5-9dab-694f8df66af5\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") "
Sep 15 13:54:22.190246 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.190117    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-metrics-certs\") pod \"df48bd0f-2591-49a5-9dab-694f8df66af5\" (UID: \"df48bd0f-2591-49a5-9dab-694f8df66af5\") "
Sep 15 13:54:22.190406 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.190384    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df48bd0f-2591-49a5-9dab-694f8df66af5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df48bd0f-2591-49a5-9dab-694f8df66af5" (UID: "df48bd0f-2591-49a5-9dab-694f8df66af5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 13:54:22.190533 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.190508    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df48bd0f-2591-49a5-9dab-694f8df66af5-openshift-service-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:54:22.192207 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.192176    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "df48bd0f-2591-49a5-9dab-694f8df66af5" (UID: "df48bd0f-2591-49a5-9dab-694f8df66af5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:54:22.192345 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.192322    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df48bd0f-2591-49a5-9dab-694f8df66af5" (UID: "df48bd0f-2591-49a5-9dab-694f8df66af5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:54:22.192451 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.192380    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df48bd0f-2591-49a5-9dab-694f8df66af5-kube-api-access-ttts5" (OuterVolumeSpecName: "kube-api-access-ttts5") pod "df48bd0f-2591-49a5-9dab-694f8df66af5" (UID: "df48bd0f-2591-49a5-9dab-694f8df66af5"). InnerVolumeSpecName "kube-api-access-ttts5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:54:22.193903 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.193880    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:22.276542 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.276499    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:22.291787 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.291756    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-maas-api-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:54:22.291787 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.291785    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttts5\" (UniqueName: \"kubernetes.io/projected/df48bd0f-2591-49a5-9dab-694f8df66af5-kube-api-access-ttts5\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:54:22.291787 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.291794    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df48bd0f-2591-49a5-9dab-694f8df66af5-metrics-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:54:22.709184 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.709131    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:22.835048 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.835016    2582 generic.go:356] "Generic (PLEG): container finished" podID="df48bd0f-2591-49a5-9dab-694f8df66af5" containerID="e82a2a89405c73a67de9fdcd483fe9704deb98a6b678e13ecdc2028aad8486a6" exitCode=0
Sep 15 13:54:22.835236 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.835075    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"
Sep 15 13:54:22.835236 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.835112    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s" event={"ID":"df48bd0f-2591-49a5-9dab-694f8df66af5","Type":"ContainerDied","Data":"e82a2a89405c73a67de9fdcd483fe9704deb98a6b678e13ecdc2028aad8486a6"}
Sep 15 13:54:22.835236 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.835179    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s" event={"ID":"df48bd0f-2591-49a5-9dab-694f8df66af5","Type":"ContainerDied","Data":"2a15d5194293cfa960a1476ff7bbec3c0997587daa529b04067606e92a728163"}
Sep 15 13:54:22.835236 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.835205    2582 scope.go:117] "RemoveContainer" containerID="e82a2a89405c73a67de9fdcd483fe9704deb98a6b678e13ecdc2028aad8486a6"
Sep 15 13:54:22.844068 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.844049    2582 scope.go:117] "RemoveContainer" containerID="e82a2a89405c73a67de9fdcd483fe9704deb98a6b678e13ecdc2028aad8486a6"
Sep 15 13:54:22.844365 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:54:22.844347    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e82a2a89405c73a67de9fdcd483fe9704deb98a6b678e13ecdc2028aad8486a6\": container with ID starting with e82a2a89405c73a67de9fdcd483fe9704deb98a6b678e13ecdc2028aad8486a6 not found: ID does not exist" containerID="e82a2a89405c73a67de9fdcd483fe9704deb98a6b678e13ecdc2028aad8486a6"
Sep 15 13:54:22.844434 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.844373    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e82a2a89405c73a67de9fdcd483fe9704deb98a6b678e13ecdc2028aad8486a6"} err="failed to get container status \"e82a2a89405c73a67de9fdcd483fe9704deb98a6b678e13ecdc2028aad8486a6\": rpc error: code = NotFound desc = could not find container \"e82a2a89405c73a67de9fdcd483fe9704deb98a6b678e13ecdc2028aad8486a6\": container with ID starting with e82a2a89405c73a67de9fdcd483fe9704deb98a6b678e13ecdc2028aad8486a6 not found: ID does not exist"
Sep 15 13:54:22.852011 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.851989    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"]
Sep 15 13:54:22.856430 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.856406    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-974680ec-84c8cf76c5-bnp7s"]
Sep 15 13:54:22.943409 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:22.943309    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:23.039783 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:23.039740    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:23.148057 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:23.148017    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:23.622461 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:23.622420    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:24.393961 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:24.393918    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:24.643868 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:24.643836    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df48bd0f-2591-49a5-9dab-694f8df66af5" path="/var/lib/kubelet/pods/df48bd0f-2591-49a5-9dab-694f8df66af5/volumes"
Sep 15 13:54:25.089691 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:25.089614    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:25.781023 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:25.780984    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:26.705573 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:26.705527    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:29.925639 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:29.925603    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:54:30.514508 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:30.514478    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:30.792591 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:30.792514    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:31.677050 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:31.677017    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:32.590965 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:32.590924    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:33.294766 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:33.294727    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:34.098084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:34.098042    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:34.815606 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:34.815568    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:35.500916 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:35.500883    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:41.341025 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.340985    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"]
Sep 15 13:54:41.341723 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.341252    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg" podUID="6354e4e1-2b19-4872-ab2c-4f404a262c05" containerName="maas-api" containerID="cri-o://e676a1c034240b93a4e830ecc3b9e795b7a39ab555350eb8dfc032beaf908d9e" gracePeriod=30
Sep 15 13:54:41.395686 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.395656    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:41.451363 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.451330    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"]
Sep 15 13:54:41.451591 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.451547    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5" podUID="95281f89-c3a2-4518-a7ff-9226aabd196f" containerName="payload-pre-processing" containerID="cri-o://ba798ef3a9ea0a710f856b4d0b4a282bbd9c1137ab60ddd5b4a8ea6480d94936" gracePeriod=30
Sep 15 13:54:41.509439 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.509404    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:41.613382 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.613327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:41.616526 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.616502    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:54:41.680633 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.680598    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84gwg\" (UniqueName: \"kubernetes.io/projected/6354e4e1-2b19-4872-ab2c-4f404a262c05-kube-api-access-84gwg\") pod \"6354e4e1-2b19-4872-ab2c-4f404a262c05\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") "
Sep 15 13:54:41.680819 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.680674    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6354e4e1-2b19-4872-ab2c-4f404a262c05-maas-api-tls\") pod \"6354e4e1-2b19-4872-ab2c-4f404a262c05\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") "
Sep 15 13:54:41.680819 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.680710    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6354e4e1-2b19-4872-ab2c-4f404a262c05-metrics-certs\") pod \"6354e4e1-2b19-4872-ab2c-4f404a262c05\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") "
Sep 15 13:54:41.680819 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.680765    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6354e4e1-2b19-4872-ab2c-4f404a262c05-openshift-service-ca\") pod \"6354e4e1-2b19-4872-ab2c-4f404a262c05\" (UID: \"6354e4e1-2b19-4872-ab2c-4f404a262c05\") "
Sep 15 13:54:41.681346 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.681308    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6354e4e1-2b19-4872-ab2c-4f404a262c05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6354e4e1-2b19-4872-ab2c-4f404a262c05" (UID: "6354e4e1-2b19-4872-ab2c-4f404a262c05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 13:54:41.682792 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.682759    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6354e4e1-2b19-4872-ab2c-4f404a262c05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6354e4e1-2b19-4872-ab2c-4f404a262c05" (UID: "6354e4e1-2b19-4872-ab2c-4f404a262c05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:54:41.682890 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.682796    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6354e4e1-2b19-4872-ab2c-4f404a262c05-kube-api-access-84gwg" (OuterVolumeSpecName: "kube-api-access-84gwg") pod "6354e4e1-2b19-4872-ab2c-4f404a262c05" (UID: "6354e4e1-2b19-4872-ab2c-4f404a262c05"). InnerVolumeSpecName "kube-api-access-84gwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:54:41.682890 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.682881    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6354e4e1-2b19-4872-ab2c-4f404a262c05-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6354e4e1-2b19-4872-ab2c-4f404a262c05" (UID: "6354e4e1-2b19-4872-ab2c-4f404a262c05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:54:41.699905 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.699879    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:41.721920 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.721893    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"
Sep 15 13:54:41.761886 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.761848    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:41.782068 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.782035    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhhh6\" (UniqueName: \"kubernetes.io/projected/95281f89-c3a2-4518-a7ff-9226aabd196f-kube-api-access-qhhh6\") pod \"95281f89-c3a2-4518-a7ff-9226aabd196f\" (UID: \"95281f89-c3a2-4518-a7ff-9226aabd196f\") "
Sep 15 13:54:41.782068 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.782071    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95281f89-c3a2-4518-a7ff-9226aabd196f-plugins-config-volume\") pod \"95281f89-c3a2-4518-a7ff-9226aabd196f\" (UID: \"95281f89-c3a2-4518-a7ff-9226aabd196f\") "
Sep 15 13:54:41.782446 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.782422    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84gwg\" (UniqueName: \"kubernetes.io/projected/6354e4e1-2b19-4872-ab2c-4f404a262c05-kube-api-access-84gwg\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:54:41.782529 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.782449    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6354e4e1-2b19-4872-ab2c-4f404a262c05-maas-api-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:54:41.782529 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.782448    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95281f89-c3a2-4518-a7ff-9226aabd196f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95281f89-c3a2-4518-a7ff-9226aabd196f" (UID: "95281f89-c3a2-4518-a7ff-9226aabd196f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 13:54:41.782529 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.782463    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6354e4e1-2b19-4872-ab2c-4f404a262c05-metrics-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:54:41.782529 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.782478    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6354e4e1-2b19-4872-ab2c-4f404a262c05-openshift-service-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:54:41.784207 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.784185    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95281f89-c3a2-4518-a7ff-9226aabd196f-kube-api-access-qhhh6" (OuterVolumeSpecName: "kube-api-access-qhhh6") pod "95281f89-c3a2-4518-a7ff-9226aabd196f" (UID: "95281f89-c3a2-4518-a7ff-9226aabd196f"). InnerVolumeSpecName "kube-api-access-qhhh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:54:41.883737 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.883653    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhhh6\" (UniqueName: \"kubernetes.io/projected/95281f89-c3a2-4518-a7ff-9226aabd196f-kube-api-access-qhhh6\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:54:41.883737 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.883682    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95281f89-c3a2-4518-a7ff-9226aabd196f-plugins-config-volume\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:54:41.939572 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.939537    2582 generic.go:356] "Generic (PLEG): container finished" podID="6354e4e1-2b19-4872-ab2c-4f404a262c05" containerID="e676a1c034240b93a4e830ecc3b9e795b7a39ab555350eb8dfc032beaf908d9e" exitCode=0
Sep 15 13:54:41.939705 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.939580    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"
Sep 15 13:54:41.939705 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.939622    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg" event={"ID":"6354e4e1-2b19-4872-ab2c-4f404a262c05","Type":"ContainerDied","Data":"e676a1c034240b93a4e830ecc3b9e795b7a39ab555350eb8dfc032beaf908d9e"}
Sep 15 13:54:41.939705 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.939671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg" event={"ID":"6354e4e1-2b19-4872-ab2c-4f404a262c05","Type":"ContainerDied","Data":"94483cda49fcc43c58edd24effc6dd93a807601f7a8ecc82576e1ae6a4752179"}
Sep 15 13:54:41.939705 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.939698    2582 scope.go:117] "RemoveContainer" containerID="e676a1c034240b93a4e830ecc3b9e795b7a39ab555350eb8dfc032beaf908d9e"
Sep 15 13:54:41.941037 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.941011    2582 generic.go:356] "Generic (PLEG): container finished" podID="95281f89-c3a2-4518-a7ff-9226aabd196f" containerID="ba798ef3a9ea0a710f856b4d0b4a282bbd9c1137ab60ddd5b4a8ea6480d94936" exitCode=0
Sep 15 13:54:41.941183 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.941059    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"
Sep 15 13:54:41.941183 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.941091    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5" event={"ID":"95281f89-c3a2-4518-a7ff-9226aabd196f","Type":"ContainerDied","Data":"ba798ef3a9ea0a710f856b4d0b4a282bbd9c1137ab60ddd5b4a8ea6480d94936"}
Sep 15 13:54:41.941287 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.941189    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5" event={"ID":"95281f89-c3a2-4518-a7ff-9226aabd196f","Type":"ContainerDied","Data":"455f274829fc56e025a5521a50e523a95a7ee857315964c7001015e17fdf2b63"}
Sep 15 13:54:41.950886 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.950870    2582 scope.go:117] "RemoveContainer" containerID="e676a1c034240b93a4e830ecc3b9e795b7a39ab555350eb8dfc032beaf908d9e"
Sep 15 13:54:41.951169 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:54:41.951134    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e676a1c034240b93a4e830ecc3b9e795b7a39ab555350eb8dfc032beaf908d9e\": container with ID starting with e676a1c034240b93a4e830ecc3b9e795b7a39ab555350eb8dfc032beaf908d9e not found: ID does not exist" containerID="e676a1c034240b93a4e830ecc3b9e795b7a39ab555350eb8dfc032beaf908d9e"
Sep 15 13:54:41.951270 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.951181    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e676a1c034240b93a4e830ecc3b9e795b7a39ab555350eb8dfc032beaf908d9e"} err="failed to get container status \"e676a1c034240b93a4e830ecc3b9e795b7a39ab555350eb8dfc032beaf908d9e\": rpc error: code = NotFound desc = could not find container \"e676a1c034240b93a4e830ecc3b9e795b7a39ab555350eb8dfc032beaf908d9e\": container with ID starting with e676a1c034240b93a4e830ecc3b9e795b7a39ab555350eb8dfc032beaf908d9e not found: ID does not exist"
Sep 15 13:54:41.951270 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.951203    2582 scope.go:117] "RemoveContainer" containerID="ba798ef3a9ea0a710f856b4d0b4a282bbd9c1137ab60ddd5b4a8ea6480d94936"
Sep 15 13:54:41.959113 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.959093    2582 scope.go:117] "RemoveContainer" containerID="ba798ef3a9ea0a710f856b4d0b4a282bbd9c1137ab60ddd5b4a8ea6480d94936"
Sep 15 13:54:41.959444 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:54:41.959424    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba798ef3a9ea0a710f856b4d0b4a282bbd9c1137ab60ddd5b4a8ea6480d94936\": container with ID starting with ba798ef3a9ea0a710f856b4d0b4a282bbd9c1137ab60ddd5b4a8ea6480d94936 not found: ID does not exist" containerID="ba798ef3a9ea0a710f856b4d0b4a282bbd9c1137ab60ddd5b4a8ea6480d94936"
Sep 15 13:54:41.959488 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.959453    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba798ef3a9ea0a710f856b4d0b4a282bbd9c1137ab60ddd5b4a8ea6480d94936"} err="failed to get container status \"ba798ef3a9ea0a710f856b4d0b4a282bbd9c1137ab60ddd5b4a8ea6480d94936\": rpc error: code = NotFound desc = could not find container \"ba798ef3a9ea0a710f856b4d0b4a282bbd9c1137ab60ddd5b4a8ea6480d94936\": container with ID starting with ba798ef3a9ea0a710f856b4d0b4a282bbd9c1137ab60ddd5b4a8ea6480d94936 not found: ID does not exist"
Sep 15 13:54:41.967403 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.967368    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"]
Sep 15 13:54:41.974964 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.974939    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-80a7cc-75fd9ff4-k4qs5"]
Sep 15 13:54:41.984648 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.984628    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"]
Sep 15 13:54:41.988560 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:41.988541    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-80a7cc-6d6f68784f-25cxg"]
Sep 15 13:54:42.051537 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:42.051505    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:42.114250 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:42.114209    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:42.643121 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:42.643045    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6354e4e1-2b19-4872-ab2c-4f404a262c05" path="/var/lib/kubelet/pods/6354e4e1-2b19-4872-ab2c-4f404a262c05/volumes"
Sep 15 13:54:42.643491 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:42.643395    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95281f89-c3a2-4518-a7ff-9226aabd196f" path="/var/lib/kubelet/pods/95281f89-c3a2-4518-a7ff-9226aabd196f/volumes"
Sep 15 13:54:42.781878 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:42.781843    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:43.720259 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:43.720223    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:44.480951 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:44.480871    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:45.232693 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:45.232664    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:47.024786 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:47.024748    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"]
Sep 15 13:54:47.119564 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:47.119531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:48.918083 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:48.918053    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:54:54.375072 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:54.375036    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:54:59.456281 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:54:59.456221    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:00.343182 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:00.343132    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:02.044056 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.044012    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh" podUID="eed098ef-4421-4022-8876-a148541b2395" containerName="main" containerID="cri-o://5ae650b678154bee204884c107d40d956ad935ecd278a25bacb165b093204b2c" gracePeriod=45
Sep 15 13:55:02.310153 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.310113    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:55:02.378734 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.378701    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgf2t\" (UniqueName: \"kubernetes.io/projected/eed098ef-4421-4022-8876-a148541b2395-kube-api-access-kgf2t\") pod \"eed098ef-4421-4022-8876-a148541b2395\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") "
Sep 15 13:55:02.378942 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.378749    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-home\") pod \"eed098ef-4421-4022-8876-a148541b2395\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") "
Sep 15 13:55:02.378942 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.378805    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed098ef-4421-4022-8876-a148541b2395-tls-certs\") pod \"eed098ef-4421-4022-8876-a148541b2395\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") "
Sep 15 13:55:02.378942 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.378823    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-tmp-dir\") pod \"eed098ef-4421-4022-8876-a148541b2395\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") "
Sep 15 13:55:02.378942 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.378848    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-model-cache\") pod \"eed098ef-4421-4022-8876-a148541b2395\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") "
Sep 15 13:55:02.378942 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.378888    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-dshm\") pod \"eed098ef-4421-4022-8876-a148541b2395\" (UID: \"eed098ef-4421-4022-8876-a148541b2395\") "
Sep 15 13:55:02.379240 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.379119    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eed098ef-4421-4022-8876-a148541b2395" (UID: "eed098ef-4421-4022-8876-a148541b2395"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:55:02.379240 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.379163    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-home" (OuterVolumeSpecName: "home") pod "eed098ef-4421-4022-8876-a148541b2395" (UID: "eed098ef-4421-4022-8876-a148541b2395"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:55:02.379240 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.379172    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-model-cache" (OuterVolumeSpecName: "model-cache") pod "eed098ef-4421-4022-8876-a148541b2395" (UID: "eed098ef-4421-4022-8876-a148541b2395"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:55:02.379349 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.379301    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-tmp-dir\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:55:02.379349 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.379316    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-model-cache\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:55:02.379349 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.379324    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-home\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:55:02.381071 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.381043    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-dshm" (OuterVolumeSpecName: "dshm") pod "eed098ef-4421-4022-8876-a148541b2395" (UID: "eed098ef-4421-4022-8876-a148541b2395"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:55:02.381200 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.381109    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed098ef-4421-4022-8876-a148541b2395-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eed098ef-4421-4022-8876-a148541b2395" (UID: "eed098ef-4421-4022-8876-a148541b2395"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:55:02.381200 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.381172    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed098ef-4421-4022-8876-a148541b2395-kube-api-access-kgf2t" (OuterVolumeSpecName: "kube-api-access-kgf2t") pod "eed098ef-4421-4022-8876-a148541b2395" (UID: "eed098ef-4421-4022-8876-a148541b2395"). InnerVolumeSpecName "kube-api-access-kgf2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:55:02.480664 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.480629    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgf2t\" (UniqueName: \"kubernetes.io/projected/eed098ef-4421-4022-8876-a148541b2395-kube-api-access-kgf2t\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:55:02.480664 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.480659    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed098ef-4421-4022-8876-a148541b2395-tls-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:55:02.480664 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:02.480668    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed098ef-4421-4022-8876-a148541b2395-dshm\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:55:03.063294 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:03.063202    2582 generic.go:356] "Generic (PLEG): container finished" podID="eed098ef-4421-4022-8876-a148541b2395" containerID="5ae650b678154bee204884c107d40d956ad935ecd278a25bacb165b093204b2c" exitCode=0
Sep 15 13:55:03.063294 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:03.063239    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh" event={"ID":"eed098ef-4421-4022-8876-a148541b2395","Type":"ContainerDied","Data":"5ae650b678154bee204884c107d40d956ad935ecd278a25bacb165b093204b2c"}
Sep 15 13:55:03.063294 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:03.063265    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"
Sep 15 13:55:03.063837 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:03.063296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh" event={"ID":"eed098ef-4421-4022-8876-a148541b2395","Type":"ContainerDied","Data":"309cba632ae6a33a20251713d31f2fa81eebb85103867ea6a456a58cd5708518"}
Sep 15 13:55:03.063837 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:03.063319    2582 scope.go:117] "RemoveContainer" containerID="5ae650b678154bee204884c107d40d956ad935ecd278a25bacb165b093204b2c"
Sep 15 13:55:03.072500 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:03.072475    2582 scope.go:117] "RemoveContainer" containerID="5ae650b678154bee204884c107d40d956ad935ecd278a25bacb165b093204b2c"
Sep 15 13:55:03.072820 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:55:03.072796    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae650b678154bee204884c107d40d956ad935ecd278a25bacb165b093204b2c\": container with ID starting with 5ae650b678154bee204884c107d40d956ad935ecd278a25bacb165b093204b2c not found: ID does not exist" containerID="5ae650b678154bee204884c107d40d956ad935ecd278a25bacb165b093204b2c"
Sep 15 13:55:03.072919 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:03.072829    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae650b678154bee204884c107d40d956ad935ecd278a25bacb165b093204b2c"} err="failed to get container status \"5ae650b678154bee204884c107d40d956ad935ecd278a25bacb165b093204b2c\": rpc error: code = NotFound desc = could not find container \"5ae650b678154bee204884c107d40d956ad935ecd278a25bacb165b093204b2c\": container with ID starting with 5ae650b678154bee204884c107d40d956ad935ecd278a25bacb165b093204b2c not found: ID does not exist"
Sep 15 13:55:03.079843 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:03.079812    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"]
Sep 15 13:55:03.083909 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:03.083887    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-80a7cc/autores-model-80a7cc-kserve-5446f869c5-hmrqh"]
Sep 15 13:55:04.020092 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:04.020055    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:55:04.643337 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:04.643301    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed098ef-4421-4022-8876-a148541b2395" path="/var/lib/kubelet/pods/eed098ef-4421-4022-8876-a148541b2395/volumes"
Sep 15 13:55:18.610326 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.610286    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"]
Sep 15 13:55:18.610913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.610686    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df48bd0f-2591-49a5-9dab-694f8df66af5" containerName="maas-api"
Sep 15 13:55:18.610913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.610701    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="df48bd0f-2591-49a5-9dab-694f8df66af5" containerName="maas-api"
Sep 15 13:55:18.610913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.610722    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed098ef-4421-4022-8876-a148541b2395" containerName="main"
Sep 15 13:55:18.610913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.610728    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed098ef-4421-4022-8876-a148541b2395" containerName="main"
Sep 15 13:55:18.610913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.610736    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95281f89-c3a2-4518-a7ff-9226aabd196f" containerName="payload-pre-processing"
Sep 15 13:55:18.610913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.610742    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95281f89-c3a2-4518-a7ff-9226aabd196f" containerName="payload-pre-processing"
Sep 15 13:55:18.610913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.610748    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6354e4e1-2b19-4872-ab2c-4f404a262c05" containerName="maas-api"
Sep 15 13:55:18.610913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.610753    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6354e4e1-2b19-4872-ab2c-4f404a262c05" containerName="maas-api"
Sep 15 13:55:18.610913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.610821    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="95281f89-c3a2-4518-a7ff-9226aabd196f" containerName="payload-pre-processing"
Sep 15 13:55:18.610913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.610829    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6354e4e1-2b19-4872-ab2c-4f404a262c05" containerName="maas-api"
Sep 15 13:55:18.610913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.610835    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="df48bd0f-2591-49a5-9dab-694f8df66af5" containerName="maas-api"
Sep 15 13:55:18.610913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.610841    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed098ef-4421-4022-8876-a148541b2395" containerName="main"
Sep 15 13:55:18.615626 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.615605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.618227 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.618203    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1293e7\"/\"premium-w0-1293e7-kserve-self-signed-certs\""
Sep 15 13:55:18.625397 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.625371    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"]
Sep 15 13:55:18.647987 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.647957    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:18.735113 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.735076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8hlg\" (UniqueName: \"kubernetes.io/projected/cbd031cb-8fa1-458c-a502-f1b39f1632ee-kube-api-access-l8hlg\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.735113 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.735110    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-model-cache\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.735398 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.735128    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd031cb-8fa1-458c-a502-f1b39f1632ee-tls-certs\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.735398 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.735281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-home\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.735398 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.735304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-dshm\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.735567 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.735533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-tmp-dir\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.836770 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.836734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-home\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.836770 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.836770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-dshm\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.837021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.836890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-tmp-dir\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.837021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.836934    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8hlg\" (UniqueName: \"kubernetes.io/projected/cbd031cb-8fa1-458c-a502-f1b39f1632ee-kube-api-access-l8hlg\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.837021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.836965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-model-cache\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.837021 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.836992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd031cb-8fa1-458c-a502-f1b39f1632ee-tls-certs\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.837366 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.837302    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-home\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.837414 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.837393    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-model-cache\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.837529 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.837509    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-tmp-dir\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.839242 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.839210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-dshm\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.839461 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.839444    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd031cb-8fa1-458c-a502-f1b39f1632ee-tls-certs\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.845380 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.845358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8hlg\" (UniqueName: \"kubernetes.io/projected/cbd031cb-8fa1-458c-a502-f1b39f1632ee-kube-api-access-l8hlg\") pod \"premium-w0-1293e7-kserve-cd585c4b5-lkhng\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") " pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:18.927037 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:18.926938    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:19.200889 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:19.200805    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"]
Sep 15 13:55:19.203859 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:55:19.203815    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbd031cb_8fa1_458c_a502_f1b39f1632ee.slice/crio-7a0fdb055f85f59ec4240d238fa9c55fafa955f7ddb10a5bb691ba1bde0d016f WatchSource:0}: Error finding container 7a0fdb055f85f59ec4240d238fa9c55fafa955f7ddb10a5bb691ba1bde0d016f: Status 404 returned error can't find the container with id 7a0fdb055f85f59ec4240d238fa9c55fafa955f7ddb10a5bb691ba1bde0d016f
Sep 15 13:55:20.160075 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:20.160027    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng" event={"ID":"cbd031cb-8fa1-458c-a502-f1b39f1632ee","Type":"ContainerStarted","Data":"bbb5bfc23758e39a6efdf22049daa61d236feb62a7b6a33bd1d6a66f4d2f8ea8"}
Sep 15 13:55:20.160505 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:20.160083    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng" event={"ID":"cbd031cb-8fa1-458c-a502-f1b39f1632ee","Type":"ContainerStarted","Data":"7a0fdb055f85f59ec4240d238fa9c55fafa955f7ddb10a5bb691ba1bde0d016f"}
Sep 15 13:55:20.180910 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:20.180857    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng" podStartSLOduration=2.180842179 podStartE2EDuration="2.180842179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:55:20.178751349 +0000 UTC m=+1018.056654193" watchObservedRunningTime="2026-09-15 13:55:20.180842179 +0000 UTC m=+1018.058744998"
Sep 15 13:55:23.900003 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:23.899963    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:24.147293 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:24.147262    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:24.584401 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:24.584367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:24.851466 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:24.851382    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:55:24.910929 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:24.910894    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:25.071411 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:25.071375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:27.386708 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:27.386665    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:28.015326 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:28.015284    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:28.118897 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:28.118866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:28.215830 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:28.215794    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:28.927209 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:28.927169    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:28.927603 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:28.927251    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:28.939798 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:28.939771    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:29.107497 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:29.107440    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:29.224479 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:29.224378    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 13:55:29.834776 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:29.834734    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:30.701693 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:30.701659    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:31.410728 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:31.410689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:31.473927 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:31.473892    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:31.674808 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:31.674724    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:31.998500 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:31.998413    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:32.087802 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:32.087769    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:40.098941 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:40.098901    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:40.427616 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:40.427536    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:55:41.823421 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:41.823385    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"]
Sep 15 13:55:41.828330 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:41.828311    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:41.830705 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:41.830672    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-66261e\"/\"premium-w4-66261e-kserve-self-signed-certs\""
Sep 15 13:55:41.835360 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:41.835335    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"]
Sep 15 13:55:41.926946 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:41.926915    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:41.955972 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:41.955940    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-model-cache\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:41.956114 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:41.956007    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-home\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:41.956114 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:41.956053    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbprt\" (UniqueName: \"kubernetes.io/projected/992f4698-4415-4e4b-ba53-9957262d1542-kube-api-access-gbprt\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:41.956227 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:41.956107    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-dshm\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:41.956227 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:41.956172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/992f4698-4415-4e4b-ba53-9957262d1542-tls-certs\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:41.956227 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:41.956214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-tmp-dir\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.057173 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.057117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbprt\" (UniqueName: \"kubernetes.io/projected/992f4698-4415-4e4b-ba53-9957262d1542-kube-api-access-gbprt\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.057345 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.057181    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-dshm\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.057393 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.057339    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/992f4698-4415-4e4b-ba53-9957262d1542-tls-certs\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.057430 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.057404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-tmp-dir\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.057499 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.057478    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-model-cache\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.057589 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.057572    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-home\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.057799 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.057774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-tmp-dir\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.057929 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.057872    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-home\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.057929 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.057868    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-model-cache\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.059380 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.059360    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-dshm\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.059667 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.059651    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/992f4698-4415-4e4b-ba53-9957262d1542-tls-certs\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.066972 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.066946    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbprt\" (UniqueName: \"kubernetes.io/projected/992f4698-4415-4e4b-ba53-9957262d1542-kube-api-access-gbprt\") pod \"premium-w4-66261e-kserve-845549df46-t2bl8\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") " pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.138213 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.138171    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:42.408260 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:42.408234    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"]
Sep 15 13:55:42.410516 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:55:42.410489    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod992f4698_4415_4e4b_ba53_9957262d1542.slice/crio-50c2ab7447f61f36677e50e51f8207b8c3f53d57f72c415f02d7c122fc9d4860 WatchSource:0}: Error finding container 50c2ab7447f61f36677e50e51f8207b8c3f53d57f72c415f02d7c122fc9d4860: Status 404 returned error can't find the container with id 50c2ab7447f61f36677e50e51f8207b8c3f53d57f72c415f02d7c122fc9d4860
Sep 15 13:55:43.286079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:43.286041    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8" event={"ID":"992f4698-4415-4e4b-ba53-9957262d1542","Type":"ContainerStarted","Data":"db8798987a21d3d3e6d2eea22054a541e8fb91153f0b8c42fc4c9860a9cf9af2"}
Sep 15 13:55:43.286079 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:43.286084    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8" event={"ID":"992f4698-4415-4e4b-ba53-9957262d1542","Type":"ContainerStarted","Data":"50c2ab7447f61f36677e50e51f8207b8c3f53d57f72c415f02d7c122fc9d4860"}
Sep 15 13:55:43.308878 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:43.308818    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8" podStartSLOduration=2.308799284 podStartE2EDuration="2.308799284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:55:43.307182167 +0000 UTC m=+1041.185084983" watchObservedRunningTime="2026-09-15 13:55:43.308799284 +0000 UTC m=+1041.186702115"
Sep 15 13:55:48.434055 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.434024    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"]
Sep 15 13:55:48.437627 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.437605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.441091 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.441067    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2aa332\"/\"kube-root-ca.crt\""
Sep 15 13:55:48.441247 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.441075    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2aa332\"/\"default-dockercfg-qptg7\""
Sep 15 13:55:48.441247 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.441105    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2aa332\"/\"test-model-2aa332-kserve-self-signed-certs\""
Sep 15 13:55:48.441247 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.441118    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2aa332\"/\"openshift-service-ca.crt\""
Sep 15 13:55:48.450168 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.450129    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"]
Sep 15 13:55:48.468068 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.468012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:48.519599 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.519569    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-tmp-dir\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.519758 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.519610    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274bbc33-8215-4c20-bf10-cb03280f82ea-tls-certs\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.519758 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.519643    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-model-cache\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.519758 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.519668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6klr\" (UniqueName: \"kubernetes.io/projected/274bbc33-8215-4c20-bf10-cb03280f82ea-kube-api-access-w6klr\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.519758 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.519707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-home\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.519897 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.519797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-dshm\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.620328 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.620287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274bbc33-8215-4c20-bf10-cb03280f82ea-tls-certs\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.620542 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.620348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-model-cache\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.620542 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.620376    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6klr\" (UniqueName: \"kubernetes.io/projected/274bbc33-8215-4c20-bf10-cb03280f82ea-kube-api-access-w6klr\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.620542 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.620395    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-home\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.620542 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.620447    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-dshm\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.620542 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.620485    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-tmp-dir\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.620870 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.620847    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-tmp-dir\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.620870 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.620861    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-home\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.620974 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.620908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-model-cache\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.622592 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.622571    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-dshm\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.622853 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.622835    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274bbc33-8215-4c20-bf10-cb03280f82ea-tls-certs\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.628548 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.628517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6klr\" (UniqueName: \"kubernetes.io/projected/274bbc33-8215-4c20-bf10-cb03280f82ea-kube-api-access-w6klr\") pod \"test-model-2aa332-kserve-855d7688df-hlbln\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") " pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:48.747339 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:48.747248    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:49.021399 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:49.021372    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"]
Sep 15 13:55:49.023858 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:55:49.023827    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod274bbc33_8215_4c20_bf10_cb03280f82ea.slice/crio-9cde56f7449985ce47abcde8b82ded5caf22f898a2249ad24e322486dd44ffbd WatchSource:0}: Error finding container 9cde56f7449985ce47abcde8b82ded5caf22f898a2249ad24e322486dd44ffbd: Status 404 returned error can't find the container with id 9cde56f7449985ce47abcde8b82ded5caf22f898a2249ad24e322486dd44ffbd
Sep 15 13:55:49.318264 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:49.318176    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln" event={"ID":"274bbc33-8215-4c20-bf10-cb03280f82ea","Type":"ContainerStarted","Data":"c6dc3a11d97d678dbe577666ce84058f0a27f0636a0f230e3262ee78f88b7879"}
Sep 15 13:55:49.318264 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:49.318224    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln" event={"ID":"274bbc33-8215-4c20-bf10-cb03280f82ea","Type":"ContainerStarted","Data":"9cde56f7449985ce47abcde8b82ded5caf22f898a2249ad24e322486dd44ffbd"}
Sep 15 13:55:49.335720 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:49.335667    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln" podStartSLOduration=1.335652232 podStartE2EDuration="1.335652232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:55:49.333888173 +0000 UTC m=+1047.211790992" watchObservedRunningTime="2026-09-15 13:55:49.335652232 +0000 UTC m=+1047.213555076"
Sep 15 13:55:52.138842 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:52.138803    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:52.139261 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:52.138870    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:52.151433 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:52.151409    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:52.344322 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:52.344292    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 13:55:54.477619 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:54.477584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:55.307536 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:55.307501    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:55.992834 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:55.992788    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:56.700548 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:56.700508    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:57.453718 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:57.453689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:58.082343 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:58.082301    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:58.192881 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:58.192850    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:55:58.452702 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:58.452666    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:55:58.748478 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:58.748389    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:58.748478 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:58.748451    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:58.760927 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:58.760901    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:55:59.388037 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:55:59.388004    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:56:18.426971 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:18.426884    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:56:24.486964 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:24.486922    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:56:24.586645 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:24.586613    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:56:24.696622 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:24.696589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:56:25.032105 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:25.032073    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:56:40.733761 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:40.733720    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:56:45.729575 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.729540    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"]
Sep 15 13:56:45.733570 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.733545    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.737438 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.737413    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f9e4ef\"/\"explicit-model-f9e4ef-kserve-self-signed-certs\""
Sep 15 13:56:45.737576 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.737444    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f9e4ef\"/\"default-dockercfg-c4w6s\""
Sep 15 13:56:45.737576 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.737460    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f9e4ef\"/\"kube-root-ca.crt\""
Sep 15 13:56:45.737576 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.737454    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f9e4ef\"/\"openshift-service-ca.crt\""
Sep 15 13:56:45.744094 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.744073    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"]
Sep 15 13:56:45.758318 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.758286    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:56:45.857596 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.857553    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5bv5\" (UniqueName: \"kubernetes.io/projected/e1150a81-77c7-4625-a796-71922de6179d-kube-api-access-s5bv5\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.857767 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.857607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-model-cache\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.857767 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.857678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-home\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.857767 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.857738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-tmp-dir\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.857886 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.857766    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-dshm\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.857886 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.857785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1150a81-77c7-4625-a796-71922de6179d-tls-certs\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.958601 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.958563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5bv5\" (UniqueName: \"kubernetes.io/projected/e1150a81-77c7-4625-a796-71922de6179d-kube-api-access-s5bv5\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.958794 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.958614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-model-cache\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.958794 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.958664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-home\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.958794 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.958686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-tmp-dir\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.958794 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.958702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-dshm\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.958794 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.958718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1150a81-77c7-4625-a796-71922de6179d-tls-certs\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.959084 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.959056    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-model-cache\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.959188 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.959134    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-home\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.959256 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.959212    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-tmp-dir\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.960920 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.960894    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-dshm\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.961219 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.961202    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1150a81-77c7-4625-a796-71922de6179d-tls-certs\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:45.968657 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:45.968634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5bv5\" (UniqueName: \"kubernetes.io/projected/e1150a81-77c7-4625-a796-71922de6179d-kube-api-access-s5bv5\") pod \"explicit-model-f9e4ef-kserve-654bccbdf5-h45jc\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") " pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:46.044085 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:46.043982    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:46.317585 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:46.317559    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"]
Sep 15 13:56:46.319290 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:56:46.319260    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1150a81_77c7_4625_a796_71922de6179d.slice/crio-c928e3c26bd597c839686914e0461810818dec70b318759523697118233afb4a WatchSource:0}: Error finding container c928e3c26bd597c839686914e0461810818dec70b318759523697118233afb4a: Status 404 returned error can't find the container with id c928e3c26bd597c839686914e0461810818dec70b318759523697118233afb4a
Sep 15 13:56:46.634273 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:46.634230    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc" event={"ID":"e1150a81-77c7-4625-a796-71922de6179d","Type":"ContainerStarted","Data":"592cd150ef7d102318b05d9e62ad2d12acb21d7db35b8756678fffeb0ea184a9"}
Sep 15 13:56:46.634273 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:46.634271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc" event={"ID":"e1150a81-77c7-4625-a796-71922de6179d","Type":"ContainerStarted","Data":"c928e3c26bd597c839686914e0461810818dec70b318759523697118233afb4a"}
Sep 15 13:56:46.652113 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:46.652052    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc" podStartSLOduration=1.652036187 podStartE2EDuration="1.652036187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:56:46.649813346 +0000 UTC m=+1104.527716167" watchObservedRunningTime="2026-09-15 13:56:46.652036187 +0000 UTC m=+1104.529939046"
Sep 15 13:56:50.921293 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:50.921254    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:56:56.020577 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:56.020546    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:56:56.045090 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:56.045050    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:56.045257 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:56.045099    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:56.058043 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:56.058009    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:56.704499 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:56.704456    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:56:56.986733 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:56.986654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:56:57.188992 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:57.188958    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:56:57.291034 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:57.290952    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:56:58.425324 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:56:58.425290    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:07.524374 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:07.524338    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:10.544004 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:10.543958    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"]
Sep 15 13:57:10.565043 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:10.564999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:11.386475 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:11.386435    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:12.088996 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:12.088953    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:12.794575 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:12.794534    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:12.968451 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:12.968336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:13.516052 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:13.516006    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:14.294556 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:14.294487    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:15.046573 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:15.046537    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:16.946849 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:16.946804    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:57:20.802446 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:20.802412    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:20.903709 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:20.903675    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:21.307823 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:21.307782    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:21.407684 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:21.407650    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:21.490052 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:21.490018    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:21.588090 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:21.588049    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:21.689538 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:21.689480    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:21.976383 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:21.976279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:22.083616 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:22.083584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:22.186182 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:22.186129    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:23.070385 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:23.070351    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:24.039744 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:24.039707    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:25.290553 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.290497    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:25.563784 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.563677    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc" podUID="e1150a81-77c7-4625-a796-71922de6179d" containerName="main" containerID="cri-o://592cd150ef7d102318b05d9e62ad2d12acb21d7db35b8756678fffeb0ea184a9" gracePeriod=45
Sep 15 13:57:25.818587 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.818515    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:57:25.844870 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.844837    2582 generic.go:356] "Generic (PLEG): container finished" podID="e1150a81-77c7-4625-a796-71922de6179d" containerID="592cd150ef7d102318b05d9e62ad2d12acb21d7db35b8756678fffeb0ea184a9" exitCode=0
Sep 15 13:57:25.845058 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.844883    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"
Sep 15 13:57:25.845058 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.844906    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc" event={"ID":"e1150a81-77c7-4625-a796-71922de6179d","Type":"ContainerDied","Data":"592cd150ef7d102318b05d9e62ad2d12acb21d7db35b8756678fffeb0ea184a9"}
Sep 15 13:57:25.845058 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.844937    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc" event={"ID":"e1150a81-77c7-4625-a796-71922de6179d","Type":"ContainerDied","Data":"c928e3c26bd597c839686914e0461810818dec70b318759523697118233afb4a"}
Sep 15 13:57:25.845058 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.844954    2582 scope.go:117] "RemoveContainer" containerID="592cd150ef7d102318b05d9e62ad2d12acb21d7db35b8756678fffeb0ea184a9"
Sep 15 13:57:25.853955 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.853935    2582 scope.go:117] "RemoveContainer" containerID="592cd150ef7d102318b05d9e62ad2d12acb21d7db35b8756678fffeb0ea184a9"
Sep 15 13:57:25.854249 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:57:25.854228    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592cd150ef7d102318b05d9e62ad2d12acb21d7db35b8756678fffeb0ea184a9\": container with ID starting with 592cd150ef7d102318b05d9e62ad2d12acb21d7db35b8756678fffeb0ea184a9 not found: ID does not exist" containerID="592cd150ef7d102318b05d9e62ad2d12acb21d7db35b8756678fffeb0ea184a9"
Sep 15 13:57:25.854344 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.854255    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592cd150ef7d102318b05d9e62ad2d12acb21d7db35b8756678fffeb0ea184a9"} err="failed to get container status \"592cd150ef7d102318b05d9e62ad2d12acb21d7db35b8756678fffeb0ea184a9\": rpc error: code = NotFound desc = could not find container \"592cd150ef7d102318b05d9e62ad2d12acb21d7db35b8756678fffeb0ea184a9\": container with ID starting with 592cd150ef7d102318b05d9e62ad2d12acb21d7db35b8756678fffeb0ea184a9 not found: ID does not exist"
Sep 15 13:57:25.934949 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.934911    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-dshm\") pod \"e1150a81-77c7-4625-a796-71922de6179d\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") "
Sep 15 13:57:25.935119 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.934969    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-tmp-dir\") pod \"e1150a81-77c7-4625-a796-71922de6179d\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") "
Sep 15 13:57:25.935119 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.935012    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1150a81-77c7-4625-a796-71922de6179d-tls-certs\") pod \"e1150a81-77c7-4625-a796-71922de6179d\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") "
Sep 15 13:57:25.935119 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.935060    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5bv5\" (UniqueName: \"kubernetes.io/projected/e1150a81-77c7-4625-a796-71922de6179d-kube-api-access-s5bv5\") pod \"e1150a81-77c7-4625-a796-71922de6179d\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") "
Sep 15 13:57:25.935119 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.935107    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-model-cache\") pod \"e1150a81-77c7-4625-a796-71922de6179d\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") "
Sep 15 13:57:25.935377 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.935129    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-home\") pod \"e1150a81-77c7-4625-a796-71922de6179d\" (UID: \"e1150a81-77c7-4625-a796-71922de6179d\") "
Sep 15 13:57:25.935377 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.935296    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1150a81-77c7-4625-a796-71922de6179d" (UID: "e1150a81-77c7-4625-a796-71922de6179d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:57:25.935478 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.935436    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1150a81-77c7-4625-a796-71922de6179d" (UID: "e1150a81-77c7-4625-a796-71922de6179d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:57:25.935523 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.935479    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-home" (OuterVolumeSpecName: "home") pod "e1150a81-77c7-4625-a796-71922de6179d" (UID: "e1150a81-77c7-4625-a796-71922de6179d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:57:25.935523 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.935514    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-model-cache\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:57:25.935592 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.935528    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-tmp-dir\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:57:25.937237 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.937210    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1150a81-77c7-4625-a796-71922de6179d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1150a81-77c7-4625-a796-71922de6179d" (UID: "e1150a81-77c7-4625-a796-71922de6179d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:57:25.937381 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.937217    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-dshm" (OuterVolumeSpecName: "dshm") pod "e1150a81-77c7-4625-a796-71922de6179d" (UID: "e1150a81-77c7-4625-a796-71922de6179d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:57:25.937381 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.937294    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1150a81-77c7-4625-a796-71922de6179d-kube-api-access-s5bv5" (OuterVolumeSpecName: "kube-api-access-s5bv5") pod "e1150a81-77c7-4625-a796-71922de6179d" (UID: "e1150a81-77c7-4625-a796-71922de6179d"). InnerVolumeSpecName "kube-api-access-s5bv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:57:25.992875 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:25.992839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:26.036253 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:26.036222    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-home\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:57:26.036253 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:26.036247    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1150a81-77c7-4625-a796-71922de6179d-dshm\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:57:26.036253 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:26.036256    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1150a81-77c7-4625-a796-71922de6179d-tls-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:57:26.036499 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:26.036269    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5bv5\" (UniqueName: \"kubernetes.io/projected/e1150a81-77c7-4625-a796-71922de6179d-kube-api-access-s5bv5\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:57:26.169606 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:26.169567    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"]
Sep 15 13:57:26.172430 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:26.172402    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-f9e4ef/explicit-model-f9e4ef-kserve-654bccbdf5-h45jc"]
Sep 15 13:57:26.643082 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:26.643047    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1150a81-77c7-4625-a796-71922de6179d" path="/var/lib/kubelet/pods/e1150a81-77c7-4625-a796-71922de6179d/volumes"
Sep 15 13:57:27.689416 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:27.689382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:27.793124 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:27.793089    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:27.924012 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:27.923968    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:28.326854 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:28.326820    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:28.624496 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:28.624454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:28.792014 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:28.791974    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:29.604752 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:29.604719    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:30.281643 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:30.281611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:30.381740 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:30.381683    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:30.481129 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:30.481094    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:36.611819 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:36.611771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:36.725546 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:36.725511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:36.813755 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:36.813719    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:36.914502 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:36.914409    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:40.689244 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:40.689201    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:47.831887 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.831787    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"]
Sep 15 13:57:47.832378 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.832200    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1150a81-77c7-4625-a796-71922de6179d" containerName="main"
Sep 15 13:57:47.832378 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.832213    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1150a81-77c7-4625-a796-71922de6179d" containerName="main"
Sep 15 13:57:47.832378 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.832298    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1150a81-77c7-4625-a796-71922de6179d" containerName="main"
Sep 15 13:57:47.836656 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.836628    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:47.840246 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.840220    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-fa9465\"/\"default-dockercfg-6wjjt\""
Sep 15 13:57:47.840394 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.840257    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-fa9465\"/\"kube-root-ca.crt\""
Sep 15 13:57:47.840394 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.840310    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-fa9465\"/\"noten-model-fa9465-kserve-self-signed-certs\""
Sep 15 13:57:47.840394 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.840219    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-fa9465\"/\"openshift-service-ca.crt\""
Sep 15 13:57:47.847082 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.847056    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"]
Sep 15 13:57:47.866492 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.866464    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:47.957413 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.957381    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5bzj\" (UniqueName: \"kubernetes.io/projected/3867ccf0-0f6c-406d-a074-817bd27c9c92-kube-api-access-f5bzj\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:47.957596 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.957424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-dshm\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:47.957596 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.957445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-model-cache\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:47.957596 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.957521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-home\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:47.957596 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.957559    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3867ccf0-0f6c-406d-a074-817bd27c9c92-tls-certs\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:47.957743 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:47.957601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-tmp-dir\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.058873 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.058833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-home\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.058873 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.058883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3867ccf0-0f6c-406d-a074-817bd27c9c92-tls-certs\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.059132 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.058921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-tmp-dir\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.059132 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.058950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5bzj\" (UniqueName: \"kubernetes.io/projected/3867ccf0-0f6c-406d-a074-817bd27c9c92-kube-api-access-f5bzj\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.059132 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.058973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-dshm\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.059132 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.058992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-model-cache\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.059380 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.059350    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-home\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.059438 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.059398    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-tmp-dir\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.059479 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.059450    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-model-cache\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.061236 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.061214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-dshm\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.061490 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.061471    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3867ccf0-0f6c-406d-a074-817bd27c9c92-tls-certs\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.067918 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.067897    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5bzj\" (UniqueName: \"kubernetes.io/projected/3867ccf0-0f6c-406d-a074-817bd27c9c92-kube-api-access-f5bzj\") pod \"noten-model-fa9465-kserve-5cd6457667-wjxnn\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") " pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.147367 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.147325    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:48.421572 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.421542    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"]
Sep 15 13:57:48.423115 ip-10-0-138-252 kubenswrapper[2582]: W0915 13:57:48.423085    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3867ccf0_0f6c_406d_a074_817bd27c9c92.slice/crio-fe282c4bafc07265f0399dff3c50f5ee510708617a04d2caabe1db55e82243d7 WatchSource:0}: Error finding container fe282c4bafc07265f0399dff3c50f5ee510708617a04d2caabe1db55e82243d7: Status 404 returned error can't find the container with id fe282c4bafc07265f0399dff3c50f5ee510708617a04d2caabe1db55e82243d7
Sep 15 13:57:48.607574 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.607538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:48.712951 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.712865    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:57:48.966080 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.965988    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn" event={"ID":"3867ccf0-0f6c-406d-a074-817bd27c9c92","Type":"ContainerStarted","Data":"0094d3cdbdd278112841e555460bf4fb28f10c17c240f49c576e925f051a718b"}
Sep 15 13:57:48.966080 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.966044    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn" event={"ID":"3867ccf0-0f6c-406d-a074-817bd27c9c92","Type":"ContainerStarted","Data":"fe282c4bafc07265f0399dff3c50f5ee510708617a04d2caabe1db55e82243d7"}
Sep 15 13:57:48.984421 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:48.984374    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn" podStartSLOduration=1.984357647 podStartE2EDuration="1.984357647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:57:48.982984612 +0000 UTC m=+1166.860887453" watchObservedRunningTime="2026-09-15 13:57:48.984357647 +0000 UTC m=+1166.862260467"
Sep 15 13:57:50.025786 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:50.025746    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:57:58.147874 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:58.147831    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:58.148335 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:58.147887    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:58.160530 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:58.160504    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:57:59.031238 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:57:59.031201    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:58:08.436674 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:08.436631    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:58:14.163496 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:14.163459    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:58:16.599846 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:16.599808    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"]
Sep 15 13:58:16.939180 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:16.939067    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:58:17.040070 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:17.040031    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:58:17.140615 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:17.140578    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:58:18.522679 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:18.522650    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:58:22.131460 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:22.131415    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:58:31.621347 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.621278    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn" podUID="3867ccf0-0f6c-406d-a074-817bd27c9c92" containerName="main" containerID="cri-o://0094d3cdbdd278112841e555460bf4fb28f10c17c240f49c576e925f051a718b" gracePeriod=45
Sep 15 13:58:31.873788 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.873719    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:58:31.887312 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.887285    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-home\") pod \"3867ccf0-0f6c-406d-a074-817bd27c9c92\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") "
Sep 15 13:58:31.887452 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.887358    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-dshm\") pod \"3867ccf0-0f6c-406d-a074-817bd27c9c92\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") "
Sep 15 13:58:31.887452 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.887379    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-model-cache\") pod \"3867ccf0-0f6c-406d-a074-817bd27c9c92\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") "
Sep 15 13:58:31.887452 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.887401    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5bzj\" (UniqueName: \"kubernetes.io/projected/3867ccf0-0f6c-406d-a074-817bd27c9c92-kube-api-access-f5bzj\") pod \"3867ccf0-0f6c-406d-a074-817bd27c9c92\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") "
Sep 15 13:58:31.887452 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.887417    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3867ccf0-0f6c-406d-a074-817bd27c9c92-tls-certs\") pod \"3867ccf0-0f6c-406d-a074-817bd27c9c92\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") "
Sep 15 13:58:31.887677 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.887520    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-tmp-dir\") pod \"3867ccf0-0f6c-406d-a074-817bd27c9c92\" (UID: \"3867ccf0-0f6c-406d-a074-817bd27c9c92\") "
Sep 15 13:58:31.887677 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.887573    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-home" (OuterVolumeSpecName: "home") pod "3867ccf0-0f6c-406d-a074-817bd27c9c92" (UID: "3867ccf0-0f6c-406d-a074-817bd27c9c92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:58:31.887677 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.887634    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-model-cache" (OuterVolumeSpecName: "model-cache") pod "3867ccf0-0f6c-406d-a074-817bd27c9c92" (UID: "3867ccf0-0f6c-406d-a074-817bd27c9c92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:58:31.887831 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.887764    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3867ccf0-0f6c-406d-a074-817bd27c9c92" (UID: "3867ccf0-0f6c-406d-a074-817bd27c9c92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:58:31.887909 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.887893    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-model-cache\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:58:31.887960 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.887913    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-tmp-dir\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:58:31.887960 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.887925    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-home\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:58:31.889682 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.889653    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3867ccf0-0f6c-406d-a074-817bd27c9c92-kube-api-access-f5bzj" (OuterVolumeSpecName: "kube-api-access-f5bzj") pod "3867ccf0-0f6c-406d-a074-817bd27c9c92" (UID: "3867ccf0-0f6c-406d-a074-817bd27c9c92"). InnerVolumeSpecName "kube-api-access-f5bzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:58:31.889682 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.889670    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-dshm" (OuterVolumeSpecName: "dshm") pod "3867ccf0-0f6c-406d-a074-817bd27c9c92" (UID: "3867ccf0-0f6c-406d-a074-817bd27c9c92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:58:31.889682 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.889669    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3867ccf0-0f6c-406d-a074-817bd27c9c92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3867ccf0-0f6c-406d-a074-817bd27c9c92" (UID: "3867ccf0-0f6c-406d-a074-817bd27c9c92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:58:31.989459 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.989425    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3867ccf0-0f6c-406d-a074-817bd27c9c92-dshm\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:58:31.989459 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.989459    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5bzj\" (UniqueName: \"kubernetes.io/projected/3867ccf0-0f6c-406d-a074-817bd27c9c92-kube-api-access-f5bzj\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:58:31.989459 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:31.989470    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3867ccf0-0f6c-406d-a074-817bd27c9c92-tls-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:58:32.193028 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:32.192923    2582 generic.go:356] "Generic (PLEG): container finished" podID="3867ccf0-0f6c-406d-a074-817bd27c9c92" containerID="0094d3cdbdd278112841e555460bf4fb28f10c17c240f49c576e925f051a718b" exitCode=0
Sep 15 13:58:32.193028 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:32.192969    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"
Sep 15 13:58:32.193028 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:32.193010    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn" event={"ID":"3867ccf0-0f6c-406d-a074-817bd27c9c92","Type":"ContainerDied","Data":"0094d3cdbdd278112841e555460bf4fb28f10c17c240f49c576e925f051a718b"}
Sep 15 13:58:32.193345 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:32.193054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn" event={"ID":"3867ccf0-0f6c-406d-a074-817bd27c9c92","Type":"ContainerDied","Data":"fe282c4bafc07265f0399dff3c50f5ee510708617a04d2caabe1db55e82243d7"}
Sep 15 13:58:32.193345 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:32.193072    2582 scope.go:117] "RemoveContainer" containerID="0094d3cdbdd278112841e555460bf4fb28f10c17c240f49c576e925f051a718b"
Sep 15 13:58:32.202195 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:32.202177    2582 scope.go:117] "RemoveContainer" containerID="0094d3cdbdd278112841e555460bf4fb28f10c17c240f49c576e925f051a718b"
Sep 15 13:58:32.202493 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:58:32.202474    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0094d3cdbdd278112841e555460bf4fb28f10c17c240f49c576e925f051a718b\": container with ID starting with 0094d3cdbdd278112841e555460bf4fb28f10c17c240f49c576e925f051a718b not found: ID does not exist" containerID="0094d3cdbdd278112841e555460bf4fb28f10c17c240f49c576e925f051a718b"
Sep 15 13:58:32.202544 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:32.202503    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0094d3cdbdd278112841e555460bf4fb28f10c17c240f49c576e925f051a718b"} err="failed to get container status \"0094d3cdbdd278112841e555460bf4fb28f10c17c240f49c576e925f051a718b\": rpc error: code = NotFound desc = could not find container \"0094d3cdbdd278112841e555460bf4fb28f10c17c240f49c576e925f051a718b\": container with ID starting with 0094d3cdbdd278112841e555460bf4fb28f10c17c240f49c576e925f051a718b not found: ID does not exist"
Sep 15 13:58:32.213869 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:32.213842    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"]
Sep 15 13:58:32.216291 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:32.216270    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-fa9465/noten-model-fa9465-kserve-5cd6457667-wjxnn"]
Sep 15 13:58:32.649928 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:32.649891    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3867ccf0-0f6c-406d-a074-817bd27c9c92" path="/var/lib/kubelet/pods/3867ccf0-0f6c-406d-a074-817bd27c9c92/volumes"
Sep 15 13:58:36.924481 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:36.924439    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:58:55.118831 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:55.118798    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:58:55.801483 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:55.801448    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:58:55.952687 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:55.952643    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:58:58.951272 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:58.951216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:58:59.020902 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:59.020869    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:58:59.493753 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:59.493716    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"]
Sep 15 13:58:59.629095 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:59.629064    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:58:59.918470 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:58:59.918434    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:00.809414 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:00.809332    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:03.158817 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:03.158785    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:03.247237 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:03.247205    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:03.323368 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:03.323325    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:03.784112 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:03.784078    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:06.317363 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:06.317322    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:06.414898 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:06.414868    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:11.396357 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:11.396321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:12.239459 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:12.239431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:14.514913 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.514872    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln" podUID="274bbc33-8215-4c20-bf10-cb03280f82ea" containerName="main" containerID="cri-o://c6dc3a11d97d678dbe577666ce84058f0a27f0636a0f230e3262ee78f88b7879" gracePeriod=45
Sep 15 13:59:14.781065 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.780994    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:59:14.820891 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.820858    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:14.894039 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.894008    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:14.904130 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.904106    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6klr\" (UniqueName: \"kubernetes.io/projected/274bbc33-8215-4c20-bf10-cb03280f82ea-kube-api-access-w6klr\") pod \"274bbc33-8215-4c20-bf10-cb03280f82ea\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") "
Sep 15 13:59:14.904221 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.904179    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-dshm\") pod \"274bbc33-8215-4c20-bf10-cb03280f82ea\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") "
Sep 15 13:59:14.904260 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.904224    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-model-cache\") pod \"274bbc33-8215-4c20-bf10-cb03280f82ea\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") "
Sep 15 13:59:14.904299 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.904274    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274bbc33-8215-4c20-bf10-cb03280f82ea-tls-certs\") pod \"274bbc33-8215-4c20-bf10-cb03280f82ea\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") "
Sep 15 13:59:14.904335 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.904310    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-home\") pod \"274bbc33-8215-4c20-bf10-cb03280f82ea\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") "
Sep 15 13:59:14.904385 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.904337    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-tmp-dir\") pod \"274bbc33-8215-4c20-bf10-cb03280f82ea\" (UID: \"274bbc33-8215-4c20-bf10-cb03280f82ea\") "
Sep 15 13:59:14.904578 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.904548    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "274bbc33-8215-4c20-bf10-cb03280f82ea" (UID: "274bbc33-8215-4c20-bf10-cb03280f82ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:59:14.904712 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.904664    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-model-cache\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:59:14.904712 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.904692    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "274bbc33-8215-4c20-bf10-cb03280f82ea" (UID: "274bbc33-8215-4c20-bf10-cb03280f82ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:59:14.904818 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.904708    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-home" (OuterVolumeSpecName: "home") pod "274bbc33-8215-4c20-bf10-cb03280f82ea" (UID: "274bbc33-8215-4c20-bf10-cb03280f82ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:59:14.906602 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.906578    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274bbc33-8215-4c20-bf10-cb03280f82ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "274bbc33-8215-4c20-bf10-cb03280f82ea" (UID: "274bbc33-8215-4c20-bf10-cb03280f82ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:59:14.906720 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.906705    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-dshm" (OuterVolumeSpecName: "dshm") pod "274bbc33-8215-4c20-bf10-cb03280f82ea" (UID: "274bbc33-8215-4c20-bf10-cb03280f82ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:59:14.906792 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:14.906757    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274bbc33-8215-4c20-bf10-cb03280f82ea-kube-api-access-w6klr" (OuterVolumeSpecName: "kube-api-access-w6klr") pod "274bbc33-8215-4c20-bf10-cb03280f82ea" (UID: "274bbc33-8215-4c20-bf10-cb03280f82ea"). InnerVolumeSpecName "kube-api-access-w6klr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:59:15.005409 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.005372    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6klr\" (UniqueName: \"kubernetes.io/projected/274bbc33-8215-4c20-bf10-cb03280f82ea-kube-api-access-w6klr\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:59:15.005409 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.005403    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-dshm\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:59:15.005409 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.005414    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274bbc33-8215-4c20-bf10-cb03280f82ea-tls-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:59:15.005647 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.005423    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-home\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:59:15.005647 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.005432    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274bbc33-8215-4c20-bf10-cb03280f82ea-tmp-dir\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 13:59:15.413250 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.413215    2582 generic.go:356] "Generic (PLEG): container finished" podID="274bbc33-8215-4c20-bf10-cb03280f82ea" containerID="c6dc3a11d97d678dbe577666ce84058f0a27f0636a0f230e3262ee78f88b7879" exitCode=0
Sep 15 13:59:15.413430 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.413259    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"
Sep 15 13:59:15.413430 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.413299    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln" event={"ID":"274bbc33-8215-4c20-bf10-cb03280f82ea","Type":"ContainerDied","Data":"c6dc3a11d97d678dbe577666ce84058f0a27f0636a0f230e3262ee78f88b7879"}
Sep 15 13:59:15.413430 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.413340    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln" event={"ID":"274bbc33-8215-4c20-bf10-cb03280f82ea","Type":"ContainerDied","Data":"9cde56f7449985ce47abcde8b82ded5caf22f898a2249ad24e322486dd44ffbd"}
Sep 15 13:59:15.413430 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.413363    2582 scope.go:117] "RemoveContainer" containerID="c6dc3a11d97d678dbe577666ce84058f0a27f0636a0f230e3262ee78f88b7879"
Sep 15 13:59:15.422584 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.422563    2582 scope.go:117] "RemoveContainer" containerID="c6dc3a11d97d678dbe577666ce84058f0a27f0636a0f230e3262ee78f88b7879"
Sep 15 13:59:15.422883 ip-10-0-138-252 kubenswrapper[2582]: E0915 13:59:15.422862    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6dc3a11d97d678dbe577666ce84058f0a27f0636a0f230e3262ee78f88b7879\": container with ID starting with c6dc3a11d97d678dbe577666ce84058f0a27f0636a0f230e3262ee78f88b7879 not found: ID does not exist" containerID="c6dc3a11d97d678dbe577666ce84058f0a27f0636a0f230e3262ee78f88b7879"
Sep 15 13:59:15.422950 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.422891    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6dc3a11d97d678dbe577666ce84058f0a27f0636a0f230e3262ee78f88b7879"} err="failed to get container status \"c6dc3a11d97d678dbe577666ce84058f0a27f0636a0f230e3262ee78f88b7879\": rpc error: code = NotFound desc = could not find container \"c6dc3a11d97d678dbe577666ce84058f0a27f0636a0f230e3262ee78f88b7879\": container with ID starting with c6dc3a11d97d678dbe577666ce84058f0a27f0636a0f230e3262ee78f88b7879 not found: ID does not exist"
Sep 15 13:59:15.439207 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.439172    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"]
Sep 15 13:59:15.446054 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:15.446020    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2aa332/test-model-2aa332-kserve-855d7688df-hlbln"]
Sep 15 13:59:16.644733 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:16.644699    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="274bbc33-8215-4c20-bf10-cb03280f82ea" path="/var/lib/kubelet/pods/274bbc33-8215-4c20-bf10-cb03280f82ea/volumes"
Sep 15 13:59:20.525778 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:20.525740    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:59:40.492304 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:40.492267    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:45.993937 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:45.993899    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:47.200112 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:47.200074    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:47.291490 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:47.291451    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:47.915361 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:47.915322    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:48.219050 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:48.218961    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:48.331989 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:48.331951    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:54.014379 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:54.014340    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:54.438681 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:54.438643    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 13:59:54.790010 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:54.789929    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:54.906400 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:54.906364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:55.046517 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:55.046433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:56.075100 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:56.075063    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:57.019972 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:57.019925    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:57.803670 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:57.803634    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:57.871607 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:57.871571    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:58.047993 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:58.047957    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:58.384224 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:58.384188    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 13:59:58.481347 ip-10-0-138-252 kubenswrapper[2582]: I0915 13:59:58.481315    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:00.143008 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.142971    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52"]
Sep 15 14:00:00.143453 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.143437    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3867ccf0-0f6c-406d-a074-817bd27c9c92" containerName="main"
Sep 15 14:00:00.143498 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.143455    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3867ccf0-0f6c-406d-a074-817bd27c9c92" containerName="main"
Sep 15 14:00:00.143498 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.143467    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="274bbc33-8215-4c20-bf10-cb03280f82ea" containerName="main"
Sep 15 14:00:00.143498 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.143473    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="274bbc33-8215-4c20-bf10-cb03280f82ea" containerName="main"
Sep 15 14:00:00.143597 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.143540    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3867ccf0-0f6c-406d-a074-817bd27c9c92" containerName="main"
Sep 15 14:00:00.143597 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.143548    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="274bbc33-8215-4c20-bf10-cb03280f82ea" containerName="main"
Sep 15 14:00:00.146761 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.146739    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52"
Sep 15 14:00:00.149366 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.149342    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt"]
Sep 15 14:00:00.150283 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.150268    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzmkv\""
Sep 15 14:00:00.152773 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.152753    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt"
Sep 15 14:00:00.165406 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.165338    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52"]
Sep 15 14:00:00.168662 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.168635    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt"]
Sep 15 14:00:00.253805 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.253774    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9lz\" (UniqueName: \"kubernetes.io/projected/decad930-320d-44ac-99aa-885e98f9815a-kube-api-access-cf9lz\") pod \"maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt\" (UID: \"decad930-320d-44ac-99aa-885e98f9815a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt"
Sep 15 14:00:00.253805 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.253812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gptj\" (UniqueName: \"kubernetes.io/projected/fa7ce1e0-e84f-451c-930f-1a80e45dcc1f-kube-api-access-4gptj\") pod \"maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52\" (UID: \"fa7ce1e0-e84f-451c-930f-1a80e45dcc1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52"
Sep 15 14:00:00.355033 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.354997    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9lz\" (UniqueName: \"kubernetes.io/projected/decad930-320d-44ac-99aa-885e98f9815a-kube-api-access-cf9lz\") pod \"maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt\" (UID: \"decad930-320d-44ac-99aa-885e98f9815a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt"
Sep 15 14:00:00.355033 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.355037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gptj\" (UniqueName: \"kubernetes.io/projected/fa7ce1e0-e84f-451c-930f-1a80e45dcc1f-kube-api-access-4gptj\") pod \"maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52\" (UID: \"fa7ce1e0-e84f-451c-930f-1a80e45dcc1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52"
Sep 15 14:00:00.364121 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.364087    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gptj\" (UniqueName: \"kubernetes.io/projected/fa7ce1e0-e84f-451c-930f-1a80e45dcc1f-kube-api-access-4gptj\") pod \"maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52\" (UID: \"fa7ce1e0-e84f-451c-930f-1a80e45dcc1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52"
Sep 15 14:00:00.364905 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.364876    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9lz\" (UniqueName: \"kubernetes.io/projected/decad930-320d-44ac-99aa-885e98f9815a-kube-api-access-cf9lz\") pod \"maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt\" (UID: \"decad930-320d-44ac-99aa-885e98f9815a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt"
Sep 15 14:00:00.457800 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.457717    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52"
Sep 15 14:00:00.467846 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.467820    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt"
Sep 15 14:00:00.734591 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.734567    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52"]
Sep 15 14:00:00.736819 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:00:00.736792    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7ce1e0_e84f_451c_930f_1a80e45dcc1f.slice/crio-67afb0f0c3879d684ac1866bc2db30f81763b0c45d10af78788e324cd9be569c WatchSource:0}: Error finding container 67afb0f0c3879d684ac1866bc2db30f81763b0c45d10af78788e324cd9be569c: Status 404 returned error can't find the container with id 67afb0f0c3879d684ac1866bc2db30f81763b0c45d10af78788e324cd9be569c
Sep 15 14:00:00.738766 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.738749    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 14:00:00.748084 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:00.748062    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt"]
Sep 15 14:00:00.749780 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:00:00.749760    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecad930_320d_44ac_99aa_885e98f9815a.slice/crio-30dbbfeeab37bea73456424842a4788075de1a59d7e4c9bdf7daf3252d1d358c WatchSource:0}: Error finding container 30dbbfeeab37bea73456424842a4788075de1a59d7e4c9bdf7daf3252d1d358c: Status 404 returned error can't find the container with id 30dbbfeeab37bea73456424842a4788075de1a59d7e4c9bdf7daf3252d1d358c
Sep 15 14:00:01.645613 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:01.645578    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt" event={"ID":"decad930-320d-44ac-99aa-885e98f9815a","Type":"ContainerStarted","Data":"30dbbfeeab37bea73456424842a4788075de1a59d7e4c9bdf7daf3252d1d358c"}
Sep 15 14:00:01.646815 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:01.646789    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52" event={"ID":"fa7ce1e0-e84f-451c-930f-1a80e45dcc1f","Type":"ContainerStarted","Data":"67afb0f0c3879d684ac1866bc2db30f81763b0c45d10af78788e324cd9be569c"}
Sep 15 14:00:03.659263 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:03.659167    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52" event={"ID":"fa7ce1e0-e84f-451c-930f-1a80e45dcc1f","Type":"ContainerStarted","Data":"eb10e6e0b801181dace71eca20147c85286f8854ea2c92ab8ac2d5fd5ee5bbce"}
Sep 15 14:00:03.660742 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:03.660713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt" event={"ID":"decad930-320d-44ac-99aa-885e98f9815a","Type":"ContainerStarted","Data":"bc0199f15fe2db956625b933b21f17919b7b80a33a061c7cfd4015a0932b5d2e"}
Sep 15 14:00:03.681688 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:03.681614    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52" podStartSLOduration=1.346973951 podStartE2EDuration="3.681599454s" totalImagesPullingTime="2.334625503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:00:03.68069329 +0000 UTC m=+1301.558596109" watchObservedRunningTime="2026-09-15 14:00:03.681599454 +0000 UTC m=+1301.559502274"
Sep 15 14:00:04.670273 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:04.670233    2582 generic.go:356] "Generic (PLEG): container finished" podID="fa7ce1e0-e84f-451c-930f-1a80e45dcc1f" containerID="eb10e6e0b801181dace71eca20147c85286f8854ea2c92ab8ac2d5fd5ee5bbce" exitCode=0
Sep 15 14:00:04.670697 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:04.670332    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52" event={"ID":"fa7ce1e0-e84f-451c-930f-1a80e45dcc1f","Type":"ContainerDied","Data":"eb10e6e0b801181dace71eca20147c85286f8854ea2c92ab8ac2d5fd5ee5bbce"}
Sep 15 14:00:04.671957 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:04.671925    2582 generic.go:356] "Generic (PLEG): container finished" podID="decad930-320d-44ac-99aa-885e98f9815a" containerID="bc0199f15fe2db956625b933b21f17919b7b80a33a061c7cfd4015a0932b5d2e" exitCode=0
Sep 15 14:00:04.672117 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:04.671992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt" event={"ID":"decad930-320d-44ac-99aa-885e98f9815a","Type":"ContainerDied","Data":"bc0199f15fe2db956625b933b21f17919b7b80a33a061c7cfd4015a0932b5d2e"}
Sep 15 14:00:04.691207 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:04.691155    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt" podStartSLOduration=2.370135767 podStartE2EDuration="4.691121431s" totalImagesPullingTime="2.320985664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:00:03.705479549 +0000 UTC m=+1301.583382372" watchObservedRunningTime="2026-09-15 14:00:04.691121431 +0000 UTC m=+1302.569024251"
Sep 15 14:00:05.850556 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:05.850528    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52"
Sep 15 14:00:05.854097 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:05.854075    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt"
Sep 15 14:00:05.913963 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:05.913926    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf9lz\" (UniqueName: \"kubernetes.io/projected/decad930-320d-44ac-99aa-885e98f9815a-kube-api-access-cf9lz\") pod \"decad930-320d-44ac-99aa-885e98f9815a\" (UID: \"decad930-320d-44ac-99aa-885e98f9815a\") "
Sep 15 14:00:05.914121 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:05.913972    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gptj\" (UniqueName: \"kubernetes.io/projected/fa7ce1e0-e84f-451c-930f-1a80e45dcc1f-kube-api-access-4gptj\") pod \"fa7ce1e0-e84f-451c-930f-1a80e45dcc1f\" (UID: \"fa7ce1e0-e84f-451c-930f-1a80e45dcc1f\") "
Sep 15 14:00:05.916126 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:05.916093    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decad930-320d-44ac-99aa-885e98f9815a-kube-api-access-cf9lz" (OuterVolumeSpecName: "kube-api-access-cf9lz") pod "decad930-320d-44ac-99aa-885e98f9815a" (UID: "decad930-320d-44ac-99aa-885e98f9815a"). InnerVolumeSpecName "kube-api-access-cf9lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:00:05.916264 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:05.916133    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7ce1e0-e84f-451c-930f-1a80e45dcc1f-kube-api-access-4gptj" (OuterVolumeSpecName: "kube-api-access-4gptj") pod "fa7ce1e0-e84f-451c-930f-1a80e45dcc1f" (UID: "fa7ce1e0-e84f-451c-930f-1a80e45dcc1f"). InnerVolumeSpecName "kube-api-access-4gptj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:00:06.015562 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:06.015466    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf9lz\" (UniqueName: \"kubernetes.io/projected/decad930-320d-44ac-99aa-885e98f9815a-kube-api-access-cf9lz\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:00:06.015562 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:06.015502    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gptj\" (UniqueName: \"kubernetes.io/projected/fa7ce1e0-e84f-451c-930f-1a80e45dcc1f-kube-api-access-4gptj\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:00:06.690391 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:06.690352    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52"
Sep 15 14:00:06.690585 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:06.690390    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52" event={"ID":"fa7ce1e0-e84f-451c-930f-1a80e45dcc1f","Type":"ContainerDied","Data":"67afb0f0c3879d684ac1866bc2db30f81763b0c45d10af78788e324cd9be569c"}
Sep 15 14:00:06.690585 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:06.690472    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67afb0f0c3879d684ac1866bc2db30f81763b0c45d10af78788e324cd9be569c"
Sep 15 14:00:06.692122 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:06.692096    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt"
Sep 15 14:00:06.692122 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:06.692111    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt" event={"ID":"decad930-320d-44ac-99aa-885e98f9815a","Type":"ContainerDied","Data":"30dbbfeeab37bea73456424842a4788075de1a59d7e4c9bdf7daf3252d1d358c"}
Sep 15 14:00:06.692311 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:06.692176    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30dbbfeeab37bea73456424842a4788075de1a59d7e4c9bdf7daf3252d1d358c"
Sep 15 14:00:19.732244 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:19.732203    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:19.832811 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:19.832771    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:00:20.110332 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:20.110299    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:20.453254 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:20.453124    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:20.578778 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:20.578734    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:20.864431 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:20.864398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:21.113845 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:21.113813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:22.113228 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:22.113195    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:23.655335 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:23.655300    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:24.122763 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:24.122730    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:27.022616 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:27.022578    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:27.122820 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:27.122785    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:28.096849 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:28.096807    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:28.828051 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:28.828011    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:29.643768 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:29.643736    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:30.403894 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:30.403840    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:31.175909 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:31.175866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:37.912480 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:37.912444    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:38.011942 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:38.011907    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:38.111550 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:38.111517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:38.176490 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:38.176401    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:38.477050 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:38.476970    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:38.598562 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:38.598530    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:39.439073 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:39.439041    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:41.461442 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:41.461406    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:41.563246 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:41.563210    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:44.295862 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:44.295829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:44.370280 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:44.370248    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:44.552600 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:44.552515    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:44.637189 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:44.637157    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:00:44.868694 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:44.868639    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:44.963696 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:44.963652    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:45.072643 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:45.072609    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:51.113468 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:51.113433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:51.181122 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:51.181082    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:51.289849 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:51.289805    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:51.474821 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:51.474735    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:57.817392 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:57.817350    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:57.915054 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:57.915024    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:58.777861 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:58.777821    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:59.190526 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:59.190482    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:00:59.309774 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:00:59.309741    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:01.517804 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:01.517762    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:02.287237 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:02.287202    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:02.387437 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:02.387398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:02.484052 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:02.484019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:03.125264 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:03.125227    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:04.005009 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:04.004969    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:04.814626 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:04.814587    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:05.905728 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:05.905693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:06.009968 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:06.009934    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:18.632609 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:18.632570    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:01:27.099668 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:27.099630    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:27.200803 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:27.200773    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:27.424407 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:27.424321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:27.724451 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:27.724362    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:27.921763 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:27.921733    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:28.023812 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:28.023727    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:28.806165 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:28.806113    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:29.803479 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:29.803439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:30.371777 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:30.371730    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:30.458448 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:30.458414    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:30.721703 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:30.721596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:31.536833 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.536800    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"]
Sep 15 14:01:31.537367 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.537343    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa7ce1e0-e84f-451c-930f-1a80e45dcc1f" containerName="cleanup"
Sep 15 14:01:31.537367 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.537367    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa7ce1e0-e84f-451c-930f-1a80e45dcc1f" containerName="cleanup"
Sep 15 14:01:31.537507 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.537380    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="decad930-320d-44ac-99aa-885e98f9815a" containerName="cleanup"
Sep 15 14:01:31.537507 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.537388    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="decad930-320d-44ac-99aa-885e98f9815a" containerName="cleanup"
Sep 15 14:01:31.537507 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.537505    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa7ce1e0-e84f-451c-930f-1a80e45dcc1f" containerName="cleanup"
Sep 15 14:01:31.537672 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.537518    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="decad930-320d-44ac-99aa-885e98f9815a" containerName="cleanup"
Sep 15 14:01:31.539710 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.539692    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:31.542306 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.542284    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-4bb7b3-serving-cert\""
Sep 15 14:01:31.553481 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.553452    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"]
Sep 15 14:01:31.599041 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.599003    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:31.636381 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.636343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:31.636381 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.636381    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p7zv\" (UniqueName: \"kubernetes.io/projected/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-kube-api-access-2p7zv\") pod \"maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:31.636589 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.636450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:31.636589 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.636482    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:31.696966 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.696909    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:31.737116 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.737082    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:31.737328 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.737126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:31.737328 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.737225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:31.737328 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.737244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p7zv\" (UniqueName: \"kubernetes.io/projected/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-kube-api-access-2p7zv\") pod \"maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:31.737328 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:01:31.737270    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-serving-cert: secret "maas-api-e2e-shared-b-w2-4bb7b3-serving-cert" not found
Sep 15 14:01:31.737532 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:01:31.737354    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-maas-api-tls podName:3cfbb33b-b1f7-4089-aee2-a6130806c9dd nodeName:}" failed. No retries permitted until 2026-09-15 14:01:32.237331076 +0000 UTC m=+1390.115233874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-maas-api-tls") pod "maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42" (UID: "3cfbb33b-b1f7-4089-aee2-a6130806c9dd") : secret "maas-api-e2e-shared-b-w2-4bb7b3-serving-cert" not found
Sep 15 14:01:31.737861 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.737835    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:31.739752 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.739727    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:31.746093 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.746071    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p7zv\" (UniqueName: \"kubernetes.io/projected/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-kube-api-access-2p7zv\") pod \"maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:31.813905 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:31.813812    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:32.097409 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:32.097373    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:32.195745 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:32.195708    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:32.241055 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:32.241016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:32.243549 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:32.243516    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:32.452789 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:32.452703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:32.495251 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:32.495222    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:32.726923 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:32.726863    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"]
Sep 15 14:01:32.729364 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:01:32.729338    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfbb33b_b1f7_4089_aee2_a6130806c9dd.slice/crio-8b8fb6012c87e977bf11f393a081ba52ab567f43ae679be2fd0134f163ce48d0 WatchSource:0}: Error finding container 8b8fb6012c87e977bf11f393a081ba52ab567f43ae679be2fd0134f163ce48d0: Status 404 returned error can't find the container with id 8b8fb6012c87e977bf11f393a081ba52ab567f43ae679be2fd0134f163ce48d0
Sep 15 14:01:33.141289 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:33.141256    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42" event={"ID":"3cfbb33b-b1f7-4089-aee2-a6130806c9dd","Type":"ContainerStarted","Data":"0c7ccad562e3f5397cc0760a500acf47e0fa06e086c12e0c5fdac7bf1328c401"}
Sep 15 14:01:33.141449 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:33.141307    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42" event={"ID":"3cfbb33b-b1f7-4089-aee2-a6130806c9dd","Type":"ContainerStarted","Data":"8b8fb6012c87e977bf11f393a081ba52ab567f43ae679be2fd0134f163ce48d0"}
Sep 15 14:01:33.141449 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:33.141371    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:33.162967 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:33.162911    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42" podStartSLOduration=1.8487373919999999 podStartE2EDuration="2.16289748s" totalImagesPullingTime="314.160088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:01:33.161265083 +0000 UTC m=+1391.039167903" watchObservedRunningTime="2026-09-15 14:01:33.16289748 +0000 UTC m=+1391.040800298"
Sep 15 14:01:33.311042 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:33.310931    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:34.068787 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:34.068748    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:34.926787 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:34.926754    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:39.152182 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:39.152123    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:01:39.237068 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:39.237038    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:01:53.726038 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:53.726005    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:54.478702 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:54.478661    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:55.192787 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:55.192744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:55.291728 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:55.291695    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:56.143902 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:56.143869    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:01:56.987385 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:01:56.987331    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:01.109494 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:01.109462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:01.211068 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:01.211030    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:03.840793 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:03.840754    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:04.128978 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:04.128879    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:04.375787 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:04.375749    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:09.697122 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:09.697082    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:09.797306 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:09.797270    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:10.320320 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:10.320285    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:11.116659 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:11.116626    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:11.868467 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:11.868408    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:11.938267 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:11.938237    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:12.365518 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.365476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:12.464971 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.464931    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:12.484707 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.484675    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"]
Sep 15 14:02:12.487241 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.487218    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"
Sep 15 14:02:12.490006 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.489984    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5842bc8c\""
Sep 15 14:02:12.490273 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.490254    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5842bc8c-dockercfg-57jv2\""
Sep 15 14:02:12.500463 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.500440    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"]
Sep 15 14:02:12.627438 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.627347    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab51467-d8c4-4eb1-988e-97afe21db241-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6\" (UID: \"cab51467-d8c4-4eb1-988e-97afe21db241\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"
Sep 15 14:02:12.627580 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.627465    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktfkh\" (UniqueName: \"kubernetes.io/projected/cab51467-d8c4-4eb1-988e-97afe21db241-kube-api-access-ktfkh\") pod \"payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6\" (UID: \"cab51467-d8c4-4eb1-988e-97afe21db241\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"
Sep 15 14:02:12.728002 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.727963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab51467-d8c4-4eb1-988e-97afe21db241-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6\" (UID: \"cab51467-d8c4-4eb1-988e-97afe21db241\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"
Sep 15 14:02:12.728225 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.728135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktfkh\" (UniqueName: \"kubernetes.io/projected/cab51467-d8c4-4eb1-988e-97afe21db241-kube-api-access-ktfkh\") pod \"payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6\" (UID: \"cab51467-d8c4-4eb1-988e-97afe21db241\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"
Sep 15 14:02:12.728779 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.728755    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab51467-d8c4-4eb1-988e-97afe21db241-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6\" (UID: \"cab51467-d8c4-4eb1-988e-97afe21db241\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"
Sep 15 14:02:12.736279 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.736251    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktfkh\" (UniqueName: \"kubernetes.io/projected/cab51467-d8c4-4eb1-988e-97afe21db241-kube-api-access-ktfkh\") pod \"payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6\" (UID: \"cab51467-d8c4-4eb1-988e-97afe21db241\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"
Sep 15 14:02:12.797020 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:12.796977    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"
Sep 15 14:02:13.072423 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:13.071187    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"]
Sep 15 14:02:13.358891 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:13.358852    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6" event={"ID":"cab51467-d8c4-4eb1-988e-97afe21db241","Type":"ContainerStarted","Data":"211d911f222136427dadf3d4d05378065d21f4cc7a7e94801b392727641ecaf5"}
Sep 15 14:02:13.359088 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:13.358921    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"
Sep 15 14:02:13.359088 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:13.358947    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6" event={"ID":"cab51467-d8c4-4eb1-988e-97afe21db241","Type":"ContainerStarted","Data":"c61d5f85df563174cf4a1ea749fc262906802ca01946dde1c788c70d9b8aea27"}
Sep 15 14:02:13.377484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:13.377420    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6" podStartSLOduration=1.377399801 podStartE2EDuration="1.377399801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:02:13.375073448 +0000 UTC m=+1431.252976267" watchObservedRunningTime="2026-09-15 14:02:13.377399801 +0000 UTC m=+1431.255302623"
Sep 15 14:02:24.367018 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:24.366940    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"
Sep 15 14:02:25.122708 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:25.122671    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:26.004483 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:26.004443    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:26.824770 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:26.824728    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:02:33.797805 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:33.797766    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:34.283873 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:34.283841    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:35.005150 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:35.005097    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:35.872711 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:35.872670    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:36.493117 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:36.493082    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:37.330372 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:37.330339    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:37.654120 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:37.654080    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:38.006517 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:38.006430    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:38.881540 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:38.881507    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:41.502098 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:41.502062    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:53.129105 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.129072    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"]
Sep 15 14:02:53.131697 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.131675    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.135283 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.135259    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5842bc8c\"/\"openshift-service-ca.crt\""
Sep 15 14:02:53.135424 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.135292    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5842bc8c\"/\"kube-root-ca.crt\""
Sep 15 14:02:53.135424 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.135316    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5842bc8c\"/\"e2e-lifecycle-model-5842bc8c-kserve-self-signed-certs\""
Sep 15 14:02:53.135424 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.135337    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5842bc8c\"/\"default-dockercfg-4b79w\""
Sep 15 14:02:53.140970 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.140947    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"]
Sep 15 14:02:53.212568 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.212536    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-model-cache\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.212568 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.212578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-tmp-dir\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.212806 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.212611    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rn4k\" (UniqueName: \"kubernetes.io/projected/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-kube-api-access-7rn4k\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.212806 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.212642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-home\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.212806 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.212659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-dshm\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.212806 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.212789    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-tls-certs\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.217861 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.217830    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:02:53.313994 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.313961    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-tmp-dir\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.314219 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.314017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rn4k\" (UniqueName: \"kubernetes.io/projected/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-kube-api-access-7rn4k\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.314219 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.314055    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-home\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.314219 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.314074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-dshm\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.314219 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.314208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-tls-certs\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.314461 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.314315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-model-cache\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.314461 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.314427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-tmp-dir\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.314561 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.314471    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-home\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.314653 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.314635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-model-cache\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.316350 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.316328    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-dshm\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.316825 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.316805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-tls-certs\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.322206 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.322185    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rn4k\" (UniqueName: \"kubernetes.io/projected/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-kube-api-access-7rn4k\") pod \"e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") " pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.441874 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.441783    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:02:53.719880 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:53.719773    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"]
Sep 15 14:02:53.724987 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:02:53.724947    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod558cc7f6_d7ba_4b33_95a0_df7e35a167e2.slice/crio-584e0d0e4a2706c459378edc0fbcfcc246adc8197deff110aa2ca4e831d5a1ad WatchSource:0}: Error finding container 584e0d0e4a2706c459378edc0fbcfcc246adc8197deff110aa2ca4e831d5a1ad: Status 404 returned error can't find the container with id 584e0d0e4a2706c459378edc0fbcfcc246adc8197deff110aa2ca4e831d5a1ad
Sep 15 14:02:54.577203 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:54.577163    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67" event={"ID":"558cc7f6-d7ba-4b33-95a0-df7e35a167e2","Type":"ContainerStarted","Data":"692f013b739fc200715f2b22594fb1cba606a3a0df039938588836c46e487c33"}
Sep 15 14:02:54.577584 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:54.577210    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67" event={"ID":"558cc7f6-d7ba-4b33-95a0-df7e35a167e2","Type":"ContainerStarted","Data":"584e0d0e4a2706c459378edc0fbcfcc246adc8197deff110aa2ca4e831d5a1ad"}
Sep 15 14:02:54.595391 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:02:54.595340    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67" podStartSLOduration=1.595325156 podStartE2EDuration="1.595325156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:02:54.592719847 +0000 UTC m=+1472.470622665" watchObservedRunningTime="2026-09-15 14:02:54.595325156 +0000 UTC m=+1472.473228021"
Sep 15 14:03:01.631104 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:01.631065    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:03:03.442689 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:03.442651    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:03:03.443099 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:03.442710    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:03:03.455867 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:03.455836    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:03:03.636694 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:03.636666    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:03:26.007840 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:26.007786    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:03:30.158875 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:30.158840    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"]
Sep 15 14:03:30.159366 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:30.159070    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerName="istio-proxy" containerID="cri-o://d0fe4b57b257107510baaa2d556a7cd328dc678e5d16e5ed1a52326e3ad97dc9" gracePeriod=30
Sep 15 14:03:30.182183 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:30.182156    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:03:30.403017 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:30.402984    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:03:31.418849 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:31.418818    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:03:32.802329 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:32.802295    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:03:32.915215 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:32.915175    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:03:33.816865 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:33.816830    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:03:34.581953 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:34.581920    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:03:34.794094 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:34.794055    2582 patch_prober.go:28] interesting pod/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 14:03:34.794282 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:34.794106    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 14:03:35.354719 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.354682    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:03:35.415865 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.415840    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 14:03:35.523046 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.522943    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-podinfo\") pod \"22efe0fe-300a-4cdd-81c1-81cc8d379837\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") "
Sep 15 14:03:35.523046 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.522981    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-credential-socket\") pod \"22efe0fe-300a-4cdd-81c1-81cc8d379837\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") "
Sep 15 14:03:35.523046 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.522999    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-data\") pod \"22efe0fe-300a-4cdd-81c1-81cc8d379837\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") "
Sep 15 14:03:35.523046 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523048    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-envoy\") pod \"22efe0fe-300a-4cdd-81c1-81cc8d379837\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") "
Sep 15 14:03:35.523431 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523079    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-token\") pod \"22efe0fe-300a-4cdd-81c1-81cc8d379837\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") "
Sep 15 14:03:35.523431 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523179    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22efe0fe-300a-4cdd-81c1-81cc8d379837-istiod-ca-cert\") pod \"22efe0fe-300a-4cdd-81c1-81cc8d379837\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") "
Sep 15 14:03:35.523431 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523208    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-workload-certs\") pod \"22efe0fe-300a-4cdd-81c1-81cc8d379837\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") "
Sep 15 14:03:35.523431 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523237    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-workload-socket\") pod \"22efe0fe-300a-4cdd-81c1-81cc8d379837\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") "
Sep 15 14:03:35.523431 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523271    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfhzk\" (UniqueName: \"kubernetes.io/projected/22efe0fe-300a-4cdd-81c1-81cc8d379837-kube-api-access-lfhzk\") pod \"22efe0fe-300a-4cdd-81c1-81cc8d379837\" (UID: \"22efe0fe-300a-4cdd-81c1-81cc8d379837\") "
Sep 15 14:03:35.523431 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523284    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22efe0fe-300a-4cdd-81c1-81cc8d379837" (UID: "22efe0fe-300a-4cdd-81c1-81cc8d379837"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:03:35.523431 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523384    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-data" (OuterVolumeSpecName: "istio-data") pod "22efe0fe-300a-4cdd-81c1-81cc8d379837" (UID: "22efe0fe-300a-4cdd-81c1-81cc8d379837"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:03:35.523753 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523553    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22efe0fe-300a-4cdd-81c1-81cc8d379837" (UID: "22efe0fe-300a-4cdd-81c1-81cc8d379837"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:03:35.523753 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523669    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-workload-socket\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:03:35.523753 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523699    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-credential-socket\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:03:35.523753 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523706    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22efe0fe-300a-4cdd-81c1-81cc8d379837" (UID: "22efe0fe-300a-4cdd-81c1-81cc8d379837"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:03:35.523753 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523717    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-data\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:03:35.523753 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.523744    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22efe0fe-300a-4cdd-81c1-81cc8d379837-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22efe0fe-300a-4cdd-81c1-81cc8d379837" (UID: "22efe0fe-300a-4cdd-81c1-81cc8d379837"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 14:03:35.525585 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.525558    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-token" (OuterVolumeSpecName: "istio-token") pod "22efe0fe-300a-4cdd-81c1-81cc8d379837" (UID: "22efe0fe-300a-4cdd-81c1-81cc8d379837"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:03:35.525697 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.525614    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22efe0fe-300a-4cdd-81c1-81cc8d379837" (UID: "22efe0fe-300a-4cdd-81c1-81cc8d379837"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 14:03:35.525855 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.525829    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22efe0fe-300a-4cdd-81c1-81cc8d379837" (UID: "22efe0fe-300a-4cdd-81c1-81cc8d379837"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:03:35.525955 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.525857    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22efe0fe-300a-4cdd-81c1-81cc8d379837-kube-api-access-lfhzk" (OuterVolumeSpecName: "kube-api-access-lfhzk") pod "22efe0fe-300a-4cdd-81c1-81cc8d379837" (UID: "22efe0fe-300a-4cdd-81c1-81cc8d379837"). InnerVolumeSpecName "kube-api-access-lfhzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:03:35.624273 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.624241    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-token\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:03:35.624273 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.624266    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22efe0fe-300a-4cdd-81c1-81cc8d379837-istiod-ca-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:03:35.624273 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.624275    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-workload-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:03:35.624509 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.624283    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfhzk\" (UniqueName: \"kubernetes.io/projected/22efe0fe-300a-4cdd-81c1-81cc8d379837-kube-api-access-lfhzk\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:03:35.624509 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.624294    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-podinfo\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:03:35.624509 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.624302    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22efe0fe-300a-4cdd-81c1-81cc8d379837-istio-envoy\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:03:35.797831 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.797742    2582 generic.go:356] "Generic (PLEG): container finished" podID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerID="d0fe4b57b257107510baaa2d556a7cd328dc678e5d16e5ed1a52326e3ad97dc9" exitCode=0
Sep 15 14:03:35.797831 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.797789    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"
Sep 15 14:03:35.798004 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.797829    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" event={"ID":"22efe0fe-300a-4cdd-81c1-81cc8d379837","Type":"ContainerDied","Data":"d0fe4b57b257107510baaa2d556a7cd328dc678e5d16e5ed1a52326e3ad97dc9"}
Sep 15 14:03:35.798004 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.797884    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s" event={"ID":"22efe0fe-300a-4cdd-81c1-81cc8d379837","Type":"ContainerDied","Data":"567e02d88221c7bae630b0202bad60f75ff2a12568d0c14ee17bf4d19e8365c1"}
Sep 15 14:03:35.798004 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.797907    2582 scope.go:117] "RemoveContainer" containerID="d0fe4b57b257107510baaa2d556a7cd328dc678e5d16e5ed1a52326e3ad97dc9"
Sep 15 14:03:35.810165 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.810076    2582 scope.go:117] "RemoveContainer" containerID="4c95ef6011c45f057e3fcea2e0048349879112e240e87961284ea4d14d8a4a4c"
Sep 15 14:03:35.819356 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.819333    2582 scope.go:117] "RemoveContainer" containerID="d0fe4b57b257107510baaa2d556a7cd328dc678e5d16e5ed1a52326e3ad97dc9"
Sep 15 14:03:35.819632 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:03:35.819610    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fe4b57b257107510baaa2d556a7cd328dc678e5d16e5ed1a52326e3ad97dc9\": container with ID starting with d0fe4b57b257107510baaa2d556a7cd328dc678e5d16e5ed1a52326e3ad97dc9 not found: ID does not exist" containerID="d0fe4b57b257107510baaa2d556a7cd328dc678e5d16e5ed1a52326e3ad97dc9"
Sep 15 14:03:35.819711 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.819643    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fe4b57b257107510baaa2d556a7cd328dc678e5d16e5ed1a52326e3ad97dc9"} err="failed to get container status \"d0fe4b57b257107510baaa2d556a7cd328dc678e5d16e5ed1a52326e3ad97dc9\": rpc error: code = NotFound desc = could not find container \"d0fe4b57b257107510baaa2d556a7cd328dc678e5d16e5ed1a52326e3ad97dc9\": container with ID starting with d0fe4b57b257107510baaa2d556a7cd328dc678e5d16e5ed1a52326e3ad97dc9 not found: ID does not exist"
Sep 15 14:03:35.819711 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.819667    2582 scope.go:117] "RemoveContainer" containerID="4c95ef6011c45f057e3fcea2e0048349879112e240e87961284ea4d14d8a4a4c"
Sep 15 14:03:35.819926 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:03:35.819908    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c95ef6011c45f057e3fcea2e0048349879112e240e87961284ea4d14d8a4a4c\": container with ID starting with 4c95ef6011c45f057e3fcea2e0048349879112e240e87961284ea4d14d8a4a4c not found: ID does not exist" containerID="4c95ef6011c45f057e3fcea2e0048349879112e240e87961284ea4d14d8a4a4c"
Sep 15 14:03:35.819968 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.819933    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c95ef6011c45f057e3fcea2e0048349879112e240e87961284ea4d14d8a4a4c"} err="failed to get container status \"4c95ef6011c45f057e3fcea2e0048349879112e240e87961284ea4d14d8a4a4c\": rpc error: code = NotFound desc = could not find container \"4c95ef6011c45f057e3fcea2e0048349879112e240e87961284ea4d14d8a4a4c\": container with ID starting with 4c95ef6011c45f057e3fcea2e0048349879112e240e87961284ea4d14d8a4a4c not found: ID does not exist"
Sep 15 14:03:35.831058 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.831032    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"]
Sep 15 14:03:35.835930 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:35.835905    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4441a1-openshift-default-7b8dcdd768-j644s"]
Sep 15 14:03:36.131495 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:36.131461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:03:36.432940 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:36.432857    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:03:36.643037 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:36.643007    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" path="/var/lib/kubelet/pods/22efe0fe-300a-4cdd-81c1-81cc8d379837/volumes"
Sep 15 14:03:36.862522 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:36.862484    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:03:40.811458 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:40.811423    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:03:54.563454 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:03:54.563352    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:04:12.132467 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:04:12.132425    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:04:43.269208 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:04:43.269170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:04:43.337419 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:04:43.337390    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:09.888984 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:09.888947    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:10.388384 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:10.388348    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:13.147120 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:13.147044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:13.222772 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:13.222741    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:13.307396 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:13.307352    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:14.862990 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:14.862905    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"]
Sep 15 14:05:14.868519 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:14.868485    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"]
Sep 15 14:05:15.715050 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:15.715008    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:15.837527 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:15.837494    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:15.969442 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:15.969363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:16.808346 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:16.808316    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:20.391692 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:20.391657    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:29.882484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:29.882436    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng" podUID="cbd031cb-8fa1-458c-a502-f1b39f1632ee" containerName="main" containerID="cri-o://bbb5bfc23758e39a6efdf22049daa61d236feb62a7b6a33bd1d6a66f4d2f8ea8" gracePeriod=45
Sep 15 14:05:30.139592 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.139529    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 14:05:30.226488 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.226452    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-tmp-dir\") pod \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") "
Sep 15 14:05:30.226653 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.226544    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd031cb-8fa1-458c-a502-f1b39f1632ee-tls-certs\") pod \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") "
Sep 15 14:05:30.226653 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.226567    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8hlg\" (UniqueName: \"kubernetes.io/projected/cbd031cb-8fa1-458c-a502-f1b39f1632ee-kube-api-access-l8hlg\") pod \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") "
Sep 15 14:05:30.226653 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.226595    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-dshm\") pod \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") "
Sep 15 14:05:30.226773 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.226737    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-home\") pod \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") "
Sep 15 14:05:30.226773 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.226751    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cbd031cb-8fa1-458c-a502-f1b39f1632ee" (UID: "cbd031cb-8fa1-458c-a502-f1b39f1632ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:05:30.226853 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.226824    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-model-cache\") pod \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\" (UID: \"cbd031cb-8fa1-458c-a502-f1b39f1632ee\") "
Sep 15 14:05:30.226991 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.226968    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-home" (OuterVolumeSpecName: "home") pod "cbd031cb-8fa1-458c-a502-f1b39f1632ee" (UID: "cbd031cb-8fa1-458c-a502-f1b39f1632ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:05:30.227122 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.227087    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-model-cache" (OuterVolumeSpecName: "model-cache") pod "cbd031cb-8fa1-458c-a502-f1b39f1632ee" (UID: "cbd031cb-8fa1-458c-a502-f1b39f1632ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:05:30.227315 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.227296    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-home\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:05:30.227392 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.227318    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-model-cache\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:05:30.227392 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.227334    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-tmp-dir\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:05:30.228844 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.228817    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd031cb-8fa1-458c-a502-f1b39f1632ee-kube-api-access-l8hlg" (OuterVolumeSpecName: "kube-api-access-l8hlg") pod "cbd031cb-8fa1-458c-a502-f1b39f1632ee" (UID: "cbd031cb-8fa1-458c-a502-f1b39f1632ee"). InnerVolumeSpecName "kube-api-access-l8hlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:05:30.228995 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.228981    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-dshm" (OuterVolumeSpecName: "dshm") pod "cbd031cb-8fa1-458c-a502-f1b39f1632ee" (UID: "cbd031cb-8fa1-458c-a502-f1b39f1632ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:05:30.229120 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.229105    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd031cb-8fa1-458c-a502-f1b39f1632ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cbd031cb-8fa1-458c-a502-f1b39f1632ee" (UID: "cbd031cb-8fa1-458c-a502-f1b39f1632ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 14:05:30.318793 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.318737    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl" podUID="7fb9331e-e91a-44cc-abe6-7fec8bef0b71" containerName="main" containerID="cri-o://0bf084195c0ca6a6afbbff93577810e72b29695e6034acd8fb999cca14c32956" gracePeriod=45
Sep 15 14:05:30.328618 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.328591    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd031cb-8fa1-458c-a502-f1b39f1632ee-tls-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:05:30.328618 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.328620    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8hlg\" (UniqueName: \"kubernetes.io/projected/cbd031cb-8fa1-458c-a502-f1b39f1632ee-kube-api-access-l8hlg\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:05:30.328765 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.328633    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd031cb-8fa1-458c-a502-f1b39f1632ee-dshm\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:05:30.377830 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.377788    2582 generic.go:356] "Generic (PLEG): container finished" podID="cbd031cb-8fa1-458c-a502-f1b39f1632ee" containerID="bbb5bfc23758e39a6efdf22049daa61d236feb62a7b6a33bd1d6a66f4d2f8ea8" exitCode=0
Sep 15 14:05:30.378018 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.377848    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"
Sep 15 14:05:30.378018 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.377859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng" event={"ID":"cbd031cb-8fa1-458c-a502-f1b39f1632ee","Type":"ContainerDied","Data":"bbb5bfc23758e39a6efdf22049daa61d236feb62a7b6a33bd1d6a66f4d2f8ea8"}
Sep 15 14:05:30.378018 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.377913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng" event={"ID":"cbd031cb-8fa1-458c-a502-f1b39f1632ee","Type":"ContainerDied","Data":"7a0fdb055f85f59ec4240d238fa9c55fafa955f7ddb10a5bb691ba1bde0d016f"}
Sep 15 14:05:30.378018 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.377937    2582 scope.go:117] "RemoveContainer" containerID="bbb5bfc23758e39a6efdf22049daa61d236feb62a7b6a33bd1d6a66f4d2f8ea8"
Sep 15 14:05:30.400070 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.399993    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"]
Sep 15 14:05:30.406762 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.406734    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1293e7/premium-w0-1293e7-kserve-cd585c4b5-lkhng"]
Sep 15 14:05:30.429833 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.429807    2582 scope.go:117] "RemoveContainer" containerID="bbb5bfc23758e39a6efdf22049daa61d236feb62a7b6a33bd1d6a66f4d2f8ea8"
Sep 15 14:05:30.430130 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:05:30.430112    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb5bfc23758e39a6efdf22049daa61d236feb62a7b6a33bd1d6a66f4d2f8ea8\": container with ID starting with bbb5bfc23758e39a6efdf22049daa61d236feb62a7b6a33bd1d6a66f4d2f8ea8 not found: ID does not exist" containerID="bbb5bfc23758e39a6efdf22049daa61d236feb62a7b6a33bd1d6a66f4d2f8ea8"
Sep 15 14:05:30.430224 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.430153    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb5bfc23758e39a6efdf22049daa61d236feb62a7b6a33bd1d6a66f4d2f8ea8"} err="failed to get container status \"bbb5bfc23758e39a6efdf22049daa61d236feb62a7b6a33bd1d6a66f4d2f8ea8\": rpc error: code = NotFound desc = could not find container \"bbb5bfc23758e39a6efdf22049daa61d236feb62a7b6a33bd1d6a66f4d2f8ea8\": container with ID starting with bbb5bfc23758e39a6efdf22049daa61d236feb62a7b6a33bd1d6a66f4d2f8ea8 not found: ID does not exist"
Sep 15 14:05:30.556418 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.556390    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 14:05:30.631779 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.631744    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-tls-certs\") pod \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") "
Sep 15 14:05:30.631944 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.631806    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t4kd\" (UniqueName: \"kubernetes.io/projected/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-kube-api-access-2t4kd\") pod \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") "
Sep 15 14:05:30.631944 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.631839    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-home\") pod \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") "
Sep 15 14:05:30.631944 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.631876    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-dshm\") pod \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") "
Sep 15 14:05:30.631944 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.631928    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-model-cache\") pod \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") "
Sep 15 14:05:30.632181 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.631960    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-tmp-dir\") pod \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\" (UID: \"7fb9331e-e91a-44cc-abe6-7fec8bef0b71\") "
Sep 15 14:05:30.632181 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.632123    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-home" (OuterVolumeSpecName: "home") pod "7fb9331e-e91a-44cc-abe6-7fec8bef0b71" (UID: "7fb9331e-e91a-44cc-abe6-7fec8bef0b71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:05:30.632411 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.632389    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-home\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:05:30.632493 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.632420    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fb9331e-e91a-44cc-abe6-7fec8bef0b71" (UID: "7fb9331e-e91a-44cc-abe6-7fec8bef0b71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:05:30.632493 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.632480    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fb9331e-e91a-44cc-abe6-7fec8bef0b71" (UID: "7fb9331e-e91a-44cc-abe6-7fec8bef0b71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:05:30.633999 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.633976    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-dshm" (OuterVolumeSpecName: "dshm") pod "7fb9331e-e91a-44cc-abe6-7fec8bef0b71" (UID: "7fb9331e-e91a-44cc-abe6-7fec8bef0b71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:05:30.634131 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.634113    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fb9331e-e91a-44cc-abe6-7fec8bef0b71" (UID: "7fb9331e-e91a-44cc-abe6-7fec8bef0b71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 14:05:30.634332 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.634317    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-kube-api-access-2t4kd" (OuterVolumeSpecName: "kube-api-access-2t4kd") pod "7fb9331e-e91a-44cc-abe6-7fec8bef0b71" (UID: "7fb9331e-e91a-44cc-abe6-7fec8bef0b71"). InnerVolumeSpecName "kube-api-access-2t4kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:05:30.643415 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.643389    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbd031cb-8fa1-458c-a502-f1b39f1632ee" path="/var/lib/kubelet/pods/cbd031cb-8fa1-458c-a502-f1b39f1632ee/volumes"
Sep 15 14:05:30.733362 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.733319    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-tls-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:05:30.733362 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.733353    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t4kd\" (UniqueName: \"kubernetes.io/projected/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-kube-api-access-2t4kd\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:05:30.733362 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.733366    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-dshm\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:05:30.733611 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.733378    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-model-cache\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:05:30.733611 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:30.733391    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb9331e-e91a-44cc-abe6-7fec8bef0b71-tmp-dir\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:05:31.385105 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:31.385063    2582 generic.go:356] "Generic (PLEG): container finished" podID="7fb9331e-e91a-44cc-abe6-7fec8bef0b71" containerID="0bf084195c0ca6a6afbbff93577810e72b29695e6034acd8fb999cca14c32956" exitCode=0
Sep 15 14:05:31.385607 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:31.385126    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"
Sep 15 14:05:31.385607 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:31.385162    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl" event={"ID":"7fb9331e-e91a-44cc-abe6-7fec8bef0b71","Type":"ContainerDied","Data":"0bf084195c0ca6a6afbbff93577810e72b29695e6034acd8fb999cca14c32956"}
Sep 15 14:05:31.385607 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:31.385213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl" event={"ID":"7fb9331e-e91a-44cc-abe6-7fec8bef0b71","Type":"ContainerDied","Data":"507b061568fab146634cdc000d7c217cb4c7d999eef456935bc4f6abafcac935"}
Sep 15 14:05:31.385607 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:31.385231    2582 scope.go:117] "RemoveContainer" containerID="0bf084195c0ca6a6afbbff93577810e72b29695e6034acd8fb999cca14c32956"
Sep 15 14:05:31.394319 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:31.394294    2582 scope.go:117] "RemoveContainer" containerID="0bf084195c0ca6a6afbbff93577810e72b29695e6034acd8fb999cca14c32956"
Sep 15 14:05:31.394617 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:05:31.394597    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf084195c0ca6a6afbbff93577810e72b29695e6034acd8fb999cca14c32956\": container with ID starting with 0bf084195c0ca6a6afbbff93577810e72b29695e6034acd8fb999cca14c32956 not found: ID does not exist" containerID="0bf084195c0ca6a6afbbff93577810e72b29695e6034acd8fb999cca14c32956"
Sep 15 14:05:31.394680 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:31.394625    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf084195c0ca6a6afbbff93577810e72b29695e6034acd8fb999cca14c32956"} err="failed to get container status \"0bf084195c0ca6a6afbbff93577810e72b29695e6034acd8fb999cca14c32956\": rpc error: code = NotFound desc = could not find container \"0bf084195c0ca6a6afbbff93577810e72b29695e6034acd8fb999cca14c32956\": container with ID starting with 0bf084195c0ca6a6afbbff93577810e72b29695e6034acd8fb999cca14c32956 not found: ID does not exist"
Sep 15 14:05:31.403592 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:31.403561    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"]
Sep 15 14:05:31.406052 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:31.406027    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1293e7/simulator-w0-1293e7-kserve-5746489596-42tbl"]
Sep 15 14:05:32.643619 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:32.643582    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fb9331e-e91a-44cc-abe6-7fec8bef0b71" path="/var/lib/kubelet/pods/7fb9331e-e91a-44cc-abe6-7fec8bef0b71/volumes"
Sep 15 14:05:38.688023 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:38.687986    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:51.899572 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:51.899530    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:52.001284 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:52.001248    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:52.255650 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:52.255567    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:57.883759 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:57.883724    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:05:58.921617 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:58.921580    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:05:59.321704 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:05:59.321622    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:19.411830 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:19.411790    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:20.400349 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:20.400316    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:21.152918 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:21.152878    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:21.923615 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:21.923581    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:22.813789 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:22.813687    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:23.558338 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:23.558283    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:26.593261 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:26.593206    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:29.219268 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:29.219227    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:30.818620 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:30.818590    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:32.726771 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:32.726734    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:32.826693 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:32.826654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:34.898634 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:34.898597    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:34.935506 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:34.935472    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"]
Sep 15 14:06:34.935719 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:34.935681    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6" podUID="cab51467-d8c4-4eb1-988e-97afe21db241" containerName="payload-pre-processing" containerID="cri-o://211d911f222136427dadf3d4d05378065d21f4cc7a7e94801b392727641ecaf5" gracePeriod=30
Sep 15 14:06:34.973470 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:34.973442    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:35.073645 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.073609    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:35.161451 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.161357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:35.202372 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.202349    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"
Sep 15 14:06:35.249758 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.249723    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:35.252758 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.252738    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab51467-d8c4-4eb1-988e-97afe21db241-plugins-config-volume\") pod \"cab51467-d8c4-4eb1-988e-97afe21db241\" (UID: \"cab51467-d8c4-4eb1-988e-97afe21db241\") "
Sep 15 14:06:35.252883 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.252791    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktfkh\" (UniqueName: \"kubernetes.io/projected/cab51467-d8c4-4eb1-988e-97afe21db241-kube-api-access-ktfkh\") pod \"cab51467-d8c4-4eb1-988e-97afe21db241\" (UID: \"cab51467-d8c4-4eb1-988e-97afe21db241\") "
Sep 15 14:06:35.253117 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.253095    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab51467-d8c4-4eb1-988e-97afe21db241-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cab51467-d8c4-4eb1-988e-97afe21db241" (UID: "cab51467-d8c4-4eb1-988e-97afe21db241"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 14:06:35.254878 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.254857    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab51467-d8c4-4eb1-988e-97afe21db241-kube-api-access-ktfkh" (OuterVolumeSpecName: "kube-api-access-ktfkh") pod "cab51467-d8c4-4eb1-988e-97afe21db241" (UID: "cab51467-d8c4-4eb1-988e-97afe21db241"). InnerVolumeSpecName "kube-api-access-ktfkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:06:35.354233 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.354195    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab51467-d8c4-4eb1-988e-97afe21db241-plugins-config-volume\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:06:35.354233 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.354222    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktfkh\" (UniqueName: \"kubernetes.io/projected/cab51467-d8c4-4eb1-988e-97afe21db241-kube-api-access-ktfkh\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:06:35.532428 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.532347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:35.710434 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.710399    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:35.727667 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.727634    2582 generic.go:356] "Generic (PLEG): container finished" podID="cab51467-d8c4-4eb1-988e-97afe21db241" containerID="211d911f222136427dadf3d4d05378065d21f4cc7a7e94801b392727641ecaf5" exitCode=0
Sep 15 14:06:35.727826 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.727687    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"
Sep 15 14:06:35.727826 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.727687    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6" event={"ID":"cab51467-d8c4-4eb1-988e-97afe21db241","Type":"ContainerDied","Data":"211d911f222136427dadf3d4d05378065d21f4cc7a7e94801b392727641ecaf5"}
Sep 15 14:06:35.727826 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.727752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6" event={"ID":"cab51467-d8c4-4eb1-988e-97afe21db241","Type":"ContainerDied","Data":"c61d5f85df563174cf4a1ea749fc262906802ca01946dde1c788c70d9b8aea27"}
Sep 15 14:06:35.727826 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.727784    2582 scope.go:117] "RemoveContainer" containerID="211d911f222136427dadf3d4d05378065d21f4cc7a7e94801b392727641ecaf5"
Sep 15 14:06:35.737318 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.737297    2582 scope.go:117] "RemoveContainer" containerID="211d911f222136427dadf3d4d05378065d21f4cc7a7e94801b392727641ecaf5"
Sep 15 14:06:35.737596 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:06:35.737578    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211d911f222136427dadf3d4d05378065d21f4cc7a7e94801b392727641ecaf5\": container with ID starting with 211d911f222136427dadf3d4d05378065d21f4cc7a7e94801b392727641ecaf5 not found: ID does not exist" containerID="211d911f222136427dadf3d4d05378065d21f4cc7a7e94801b392727641ecaf5"
Sep 15 14:06:35.737648 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.737606    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211d911f222136427dadf3d4d05378065d21f4cc7a7e94801b392727641ecaf5"} err="failed to get container status \"211d911f222136427dadf3d4d05378065d21f4cc7a7e94801b392727641ecaf5\": rpc error: code = NotFound desc = could not find container \"211d911f222136427dadf3d4d05378065d21f4cc7a7e94801b392727641ecaf5\": container with ID starting with 211d911f222136427dadf3d4d05378065d21f4cc7a7e94801b392727641ecaf5 not found: ID does not exist"
Sep 15 14:06:35.754685 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.754655    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"]
Sep 15 14:06:35.760341 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:35.760316    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5842bc8c-678b48df9d-z7bk6"]
Sep 15 14:06:36.502748 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:36.502711    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:36.643279 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:36.643249    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cab51467-d8c4-4eb1-988e-97afe21db241" path="/var/lib/kubelet/pods/cab51467-d8c4-4eb1-988e-97afe21db241/volumes"
Sep 15 14:06:37.395671 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:37.395628    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:37.513335 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:37.513299    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:38.225185 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:38.225135    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:40.526635 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:40.526599    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"]
Sep 15 14:06:40.624810 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:40.624775    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:41.628426 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:41.628393    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:06:53.106606 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:53.106503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:06:55.545880 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.545840    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67" podUID="558cc7f6-d7ba-4b33-95a0-df7e35a167e2" containerName="main" containerID="cri-o://692f013b739fc200715f2b22594fb1cba606a3a0df039938588836c46e487c33" gracePeriod=45
Sep 15 14:06:55.815878 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.815850    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:06:55.837756 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.837723    2582 generic.go:356] "Generic (PLEG): container finished" podID="558cc7f6-d7ba-4b33-95a0-df7e35a167e2" containerID="692f013b739fc200715f2b22594fb1cba606a3a0df039938588836c46e487c33" exitCode=0
Sep 15 14:06:55.837915 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.837766    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"
Sep 15 14:06:55.837915 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.837777    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67" event={"ID":"558cc7f6-d7ba-4b33-95a0-df7e35a167e2","Type":"ContainerDied","Data":"692f013b739fc200715f2b22594fb1cba606a3a0df039938588836c46e487c33"}
Sep 15 14:06:55.837915 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.837811    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67" event={"ID":"558cc7f6-d7ba-4b33-95a0-df7e35a167e2","Type":"ContainerDied","Data":"584e0d0e4a2706c459378edc0fbcfcc246adc8197deff110aa2ca4e831d5a1ad"}
Sep 15 14:06:55.837915 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.837832    2582 scope.go:117] "RemoveContainer" containerID="692f013b739fc200715f2b22594fb1cba606a3a0df039938588836c46e487c33"
Sep 15 14:06:55.847722 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.847694    2582 scope.go:117] "RemoveContainer" containerID="692f013b739fc200715f2b22594fb1cba606a3a0df039938588836c46e487c33"
Sep 15 14:06:55.848053 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:06:55.848020    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"692f013b739fc200715f2b22594fb1cba606a3a0df039938588836c46e487c33\": container with ID starting with 692f013b739fc200715f2b22594fb1cba606a3a0df039938588836c46e487c33 not found: ID does not exist" containerID="692f013b739fc200715f2b22594fb1cba606a3a0df039938588836c46e487c33"
Sep 15 14:06:55.848153 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.848070    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"692f013b739fc200715f2b22594fb1cba606a3a0df039938588836c46e487c33"} err="failed to get container status \"692f013b739fc200715f2b22594fb1cba606a3a0df039938588836c46e487c33\": rpc error: code = NotFound desc = could not find container \"692f013b739fc200715f2b22594fb1cba606a3a0df039938588836c46e487c33\": container with ID starting with 692f013b739fc200715f2b22594fb1cba606a3a0df039938588836c46e487c33 not found: ID does not exist"
Sep 15 14:06:55.948735 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.948701    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rn4k\" (UniqueName: \"kubernetes.io/projected/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-kube-api-access-7rn4k\") pod \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") "
Sep 15 14:06:55.948735 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.948738    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-home\") pod \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") "
Sep 15 14:06:55.948988 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.948766    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-model-cache\") pod \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") "
Sep 15 14:06:55.948988 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.948782    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-dshm\") pod \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") "
Sep 15 14:06:55.948988 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.948804    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-tmp-dir\") pod \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") "
Sep 15 14:06:55.948988 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.948948    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-tls-certs\") pod \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\" (UID: \"558cc7f6-d7ba-4b33-95a0-df7e35a167e2\") "
Sep 15 14:06:55.949223 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.949102    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "558cc7f6-d7ba-4b33-95a0-df7e35a167e2" (UID: "558cc7f6-d7ba-4b33-95a0-df7e35a167e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:06:55.949223 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.949117    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-home" (OuterVolumeSpecName: "home") pod "558cc7f6-d7ba-4b33-95a0-df7e35a167e2" (UID: "558cc7f6-d7ba-4b33-95a0-df7e35a167e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:06:55.949223 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.949134    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "558cc7f6-d7ba-4b33-95a0-df7e35a167e2" (UID: "558cc7f6-d7ba-4b33-95a0-df7e35a167e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:06:55.949378 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.949362    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-model-cache\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:06:55.949417 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.949379    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-tmp-dir\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:06:55.949417 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.949390    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-home\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:06:55.951110 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.951081    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "558cc7f6-d7ba-4b33-95a0-df7e35a167e2" (UID: "558cc7f6-d7ba-4b33-95a0-df7e35a167e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 14:06:55.951236 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.951151    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-dshm" (OuterVolumeSpecName: "dshm") pod "558cc7f6-d7ba-4b33-95a0-df7e35a167e2" (UID: "558cc7f6-d7ba-4b33-95a0-df7e35a167e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:06:55.951236 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:55.951172    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-kube-api-access-7rn4k" (OuterVolumeSpecName: "kube-api-access-7rn4k") pod "558cc7f6-d7ba-4b33-95a0-df7e35a167e2" (UID: "558cc7f6-d7ba-4b33-95a0-df7e35a167e2"). InnerVolumeSpecName "kube-api-access-7rn4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:06:56.050281 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:56.050193    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rn4k\" (UniqueName: \"kubernetes.io/projected/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-kube-api-access-7rn4k\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:06:56.050281 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:56.050227    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-dshm\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:06:56.050281 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:56.050237    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/558cc7f6-d7ba-4b33-95a0-df7e35a167e2-tls-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:06:56.159341 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:56.159309    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"]
Sep 15 14:06:56.163665 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:56.163635    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5842bc8c/e2e-lifecycle-model-5842bc8c-kserve-78cf68bbcf-tzz67"]
Sep 15 14:06:56.643513 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:06:56.643481    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="558cc7f6-d7ba-4b33-95a0-df7e35a167e2" path="/var/lib/kubelet/pods/558cc7f6-d7ba-4b33-95a0-df7e35a167e2/volumes"
Sep 15 14:07:17.275016 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:07:17.274958    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:07:18.813381 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:07:18.813347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:07:23.854554 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:07:23.854507    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:07:23.955341 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:07:23.955302    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:07:24.121401 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:07:24.121317    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:07:26.737599 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:07:26.737564    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:07:46.266964 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:07:46.266927    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:01.072285 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.072235    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:01.288081 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.288041    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:01.619827 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.619790    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:01.877214 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877119    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"]
Sep 15 14:08:01.877699 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877682    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cab51467-d8c4-4eb1-988e-97afe21db241" containerName="payload-pre-processing"
Sep 15 14:08:01.877749 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877703    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cab51467-d8c4-4eb1-988e-97afe21db241" containerName="payload-pre-processing"
Sep 15 14:08:01.877749 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877732    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerName="istio-proxy"
Sep 15 14:08:01.877749 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877741    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerName="istio-proxy"
Sep 15 14:08:01.877858 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877761    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbd031cb-8fa1-458c-a502-f1b39f1632ee" containerName="main"
Sep 15 14:08:01.877858 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877770    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbd031cb-8fa1-458c-a502-f1b39f1632ee" containerName="main"
Sep 15 14:08:01.877858 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877783    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558cc7f6-d7ba-4b33-95a0-df7e35a167e2" containerName="main"
Sep 15 14:08:01.877858 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877792    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="558cc7f6-d7ba-4b33-95a0-df7e35a167e2" containerName="main"
Sep 15 14:08:01.877858 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877800    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fb9331e-e91a-44cc-abe6-7fec8bef0b71" containerName="main"
Sep 15 14:08:01.877858 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877808    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fb9331e-e91a-44cc-abe6-7fec8bef0b71" containerName="main"
Sep 15 14:08:01.877858 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877819    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerName="istio-proxy"
Sep 15 14:08:01.877858 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877826    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerName="istio-proxy"
Sep 15 14:08:01.878129 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877907    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerName="istio-proxy"
Sep 15 14:08:01.878129 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877919    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="558cc7f6-d7ba-4b33-95a0-df7e35a167e2" containerName="main"
Sep 15 14:08:01.878129 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877928    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="22efe0fe-300a-4cdd-81c1-81cc8d379837" containerName="istio-proxy"
Sep 15 14:08:01.878129 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877937    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fb9331e-e91a-44cc-abe6-7fec8bef0b71" containerName="main"
Sep 15 14:08:01.878129 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877946    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbd031cb-8fa1-458c-a502-f1b39f1632ee" containerName="main"
Sep 15 14:08:01.878129 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.877956    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cab51467-d8c4-4eb1-988e-97afe21db241" containerName="payload-pre-processing"
Sep 15 14:08:01.881329 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.881307    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:01.884296 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.884275    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-a090a0-openshift-default-dockercfg-f2tnc\""
Sep 15 14:08:01.894709 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.894684    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"]
Sep 15 14:08:01.908472 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.908436    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:01.949440 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.949403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-credential-socket\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:01.949624 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.949456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-workload-certs\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:01.949624 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.949499    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-workload-socket\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:01.949624 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.949525    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7r8w\" (UniqueName: \"kubernetes.io/projected/21b21433-d06d-4b57-9599-f93b2c09b1b8-kube-api-access-c7r8w\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:01.949624 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.949594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-token\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:01.949624 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.949619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-data\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:01.949823 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.949639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-envoy\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:01.949823 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.949653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-podinfo\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:01.949823 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:01.949691    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b21433-d06d-4b57-9599-f93b2c09b1b8-istiod-ca-cert\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.050494 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.050444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-token\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.050703 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.050507    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-data\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.050703 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.050537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-envoy\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.050703 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.050561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-podinfo\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.050885 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.050708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b21433-d06d-4b57-9599-f93b2c09b1b8-istiod-ca-cert\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.050885 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.050798    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-credential-socket\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.050885 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.050850    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-workload-certs\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.050885 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.050881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-workload-socket\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.051114 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.050917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7r8w\" (UniqueName: \"kubernetes.io/projected/21b21433-d06d-4b57-9599-f93b2c09b1b8-kube-api-access-c7r8w\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.051114 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.050970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-data\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.051255 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.051173    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-credential-socket\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.051394 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.051366    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-workload-certs\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.051463 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.051416    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-workload-socket\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.051526 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.051507    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b21433-d06d-4b57-9599-f93b2c09b1b8-istiod-ca-cert\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.052995 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.052976    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-envoy\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.053413 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.053393    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-podinfo\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.058771 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.058744    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-token\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.058975 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.058952    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7r8w\" (UniqueName: \"kubernetes.io/projected/21b21433-d06d-4b57-9599-f93b2c09b1b8-kube-api-access-c7r8w\") pod \"e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") " pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.191676 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.191592    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:02.260233 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.260175    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:02.483385 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.478717    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"]
Sep 15 14:08:02.485811 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.485764    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 14:08:02.485956 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.485860    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 14:08:02.485956 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:02.485909    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 14:08:03.179239 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:03.179199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" event={"ID":"21b21433-d06d-4b57-9599-f93b2c09b1b8","Type":"ContainerStarted","Data":"3e9f6447ebd96af86a9ca9ef5bc77d67966d08bf6a9cb9f46390ab0b179c09fa"}
Sep 15 14:08:03.179427 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:03.179270    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" event={"ID":"21b21433-d06d-4b57-9599-f93b2c09b1b8","Type":"ContainerStarted","Data":"80c707c4e65339af3138334cb92cde1ba093dc624a1b7f4d265687434a7b6489"}
Sep 15 14:08:03.191985 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:03.191949    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:03.197191 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:03.197164    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:03.219097 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:03.219033    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" podStartSLOduration=2.219012319 podStartE2EDuration="2.219012319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:08:03.216733434 +0000 UTC m=+1781.094636254" watchObservedRunningTime="2026-09-15 14:08:03.219012319 +0000 UTC m=+1781.096915138"
Sep 15 14:08:04.185125 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:04.185093    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:04.186209 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:04.186187    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:04.394757 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:04.394723    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:05.099211 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:05.099172    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:05.223032 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:05.222997    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:05.584702 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:05.584662    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:05.974395 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:05.974358    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:06.615363 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:06.615331    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:07.475973 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:07.475936    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:08.285081 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.285039    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"]
Sep 15 14:08:08.289006 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.288980    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:08.292261 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.292237    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-a090a0-serving-cert\""
Sep 15 14:08:08.298685 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.298660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8h8\" (UniqueName: \"kubernetes.io/projected/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-kube-api-access-ks8h8\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:08.298811 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.298752    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:08.298811 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.298800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:08.298937 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.298847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:08.299853 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.299820    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"]
Sep 15 14:08:08.326090 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.326055    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:08.384286 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.384244    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:08.399662 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.399623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8h8\" (UniqueName: \"kubernetes.io/projected/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-kube-api-access-ks8h8\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:08.399662 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.399674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:08.399923 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.399708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:08.399923 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.399761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:08.400133 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:08:08.400106    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-serving-cert: secret "maas-api-e2e-shared-a-w3-a090a0-serving-cert" not found
Sep 15 14:08:08.400250 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:08:08.400212    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-maas-api-tls podName:4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7 nodeName:}" failed. No retries permitted until 2026-09-15 14:08:08.900190349 +0000 UTC m=+1786.778093150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-maas-api-tls") pod "maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx" (UID: "4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7") : secret "maas-api-e2e-shared-a-w3-a090a0-serving-cert" not found
Sep 15 14:08:08.400623 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.400598    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:08.402585 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.402563    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:08.409397 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.409373    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8h8\" (UniqueName: \"kubernetes.io/projected/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-kube-api-access-ks8h8\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:08.825296 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.825261    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:08.837164 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.837115    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:08:08.904716 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.904681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:08.904899 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:08:08.904875    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-serving-cert: secret "maas-api-e2e-shared-a-w3-a090a0-serving-cert" not found
Sep 15 14:08:08.904972 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:08:08.904955    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-maas-api-tls podName:4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7 nodeName:}" failed. No retries permitted until 2026-09-15 14:08:09.904931742 +0000 UTC m=+1787.782834539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-maas-api-tls") pod "maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx" (UID: "4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7") : secret "maas-api-e2e-shared-a-w3-a090a0-serving-cert" not found
Sep 15 14:08:08.920180 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:08.920125    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:09.303504 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:09.303451    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:09.405617 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:09.405583    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:09.912724 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:09.912691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:09.915072 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:09.915050    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:10.094931 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:10.094894    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:10.103006 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:10.102981    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:10.376779 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:10.376749    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"]
Sep 15 14:08:10.378753 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:08:10.378727    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cdd651e_5c5c_440e_ba4a_26a2a5c69ac7.slice/crio-5e12ff0c1bbd2fd4e157896f4aa88603125419514f8ae1bee7e07ce5168edd6a WatchSource:0}: Error finding container 5e12ff0c1bbd2fd4e157896f4aa88603125419514f8ae1bee7e07ce5168edd6a: Status 404 returned error can't find the container with id 5e12ff0c1bbd2fd4e157896f4aa88603125419514f8ae1bee7e07ce5168edd6a
Sep 15 14:08:10.380093 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:10.380076    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 14:08:10.976919 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:10.976833    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"]
Sep 15 14:08:10.987098 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:10.987069    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"]
Sep 15 14:08:10.993402 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:10.993374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:11.062684 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:11.062650    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:11.221615 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:11.221580    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx" event={"ID":"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7","Type":"ContainerStarted","Data":"be3a6db9d84512ecf8052d78b9a8c9a97037bd2c12d19883249685709a6718eb"}
Sep 15 14:08:11.221804 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:11.221638    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx" event={"ID":"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7","Type":"ContainerStarted","Data":"5e12ff0c1bbd2fd4e157896f4aa88603125419514f8ae1bee7e07ce5168edd6a"}
Sep 15 14:08:11.221804 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:11.221667    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:11.240567 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:11.240464    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx" podStartSLOduration=2.904096435 podStartE2EDuration="3.240447769s" totalImagesPullingTime="336.351334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:08:11.240115779 +0000 UTC m=+1789.118018600" watchObservedRunningTime="2026-09-15 14:08:11.240447769 +0000 UTC m=+1789.118350587"
Sep 15 14:08:16.362948 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:16.362867    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:17.232106 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:17.232076    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:08:22.987720 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:22.987647    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:25.932288 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:25.932259    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:25.996975 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:25.996927    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8" podUID="992f4698-4415-4e4b-ba53-9957262d1542" containerName="main" containerID="cri-o://db8798987a21d3d3e6d2eea22054a541e8fb91153f0b8c42fc4c9860a9cf9af2" gracePeriod=45
Sep 15 14:08:26.007564 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.007532    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q" podUID="cac3cc33-9acb-4215-8db9-3db1bc0be41f" containerName="main" containerID="cri-o://a0ade85d7d3b5b2ca85cef044bc37fd737c05e976f803b22cdc8cab91d0b84bf" gracePeriod=45
Sep 15 14:08:26.310018 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.309893    2582 generic.go:356] "Generic (PLEG): container finished" podID="992f4698-4415-4e4b-ba53-9957262d1542" containerID="db8798987a21d3d3e6d2eea22054a541e8fb91153f0b8c42fc4c9860a9cf9af2" exitCode=0
Sep 15 14:08:26.310018 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.309926    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8" event={"ID":"992f4698-4415-4e4b-ba53-9957262d1542","Type":"ContainerDied","Data":"db8798987a21d3d3e6d2eea22054a541e8fb91153f0b8c42fc4c9860a9cf9af2"}
Sep 15 14:08:26.310018 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.309967    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8" event={"ID":"992f4698-4415-4e4b-ba53-9957262d1542","Type":"ContainerDied","Data":"50c2ab7447f61f36677e50e51f8207b8c3f53d57f72c415f02d7c122fc9d4860"}
Sep 15 14:08:26.310018 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.310001    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50c2ab7447f61f36677e50e51f8207b8c3f53d57f72c415f02d7c122fc9d4860"
Sep 15 14:08:26.312489 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.312466    2582 generic.go:356] "Generic (PLEG): container finished" podID="cac3cc33-9acb-4215-8db9-3db1bc0be41f" containerID="a0ade85d7d3b5b2ca85cef044bc37fd737c05e976f803b22cdc8cab91d0b84bf" exitCode=0
Sep 15 14:08:26.312681 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.312607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q" event={"ID":"cac3cc33-9acb-4215-8db9-3db1bc0be41f","Type":"ContainerDied","Data":"a0ade85d7d3b5b2ca85cef044bc37fd737c05e976f803b22cdc8cab91d0b84bf"}
Sep 15 14:08:26.312853 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.312791    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 14:08:26.349725 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.349699    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 14:08:26.355208 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.355183    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-dshm\") pod \"992f4698-4415-4e4b-ba53-9957262d1542\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") "
Sep 15 14:08:26.355355 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.355236    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-tmp-dir\") pod \"992f4698-4415-4e4b-ba53-9957262d1542\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") "
Sep 15 14:08:26.355355 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.355263    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-model-cache\") pod \"992f4698-4415-4e4b-ba53-9957262d1542\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") "
Sep 15 14:08:26.355355 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.355308    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbprt\" (UniqueName: \"kubernetes.io/projected/992f4698-4415-4e4b-ba53-9957262d1542-kube-api-access-gbprt\") pod \"992f4698-4415-4e4b-ba53-9957262d1542\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") "
Sep 15 14:08:26.355513 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.355421    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-home\") pod \"992f4698-4415-4e4b-ba53-9957262d1542\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") "
Sep 15 14:08:26.355513 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.355461    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/992f4698-4415-4e4b-ba53-9957262d1542-tls-certs\") pod \"992f4698-4415-4e4b-ba53-9957262d1542\" (UID: \"992f4698-4415-4e4b-ba53-9957262d1542\") "
Sep 15 14:08:26.355616 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.355526    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "992f4698-4415-4e4b-ba53-9957262d1542" (UID: "992f4698-4415-4e4b-ba53-9957262d1542"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:08:26.355616 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.355582    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-model-cache" (OuterVolumeSpecName: "model-cache") pod "992f4698-4415-4e4b-ba53-9957262d1542" (UID: "992f4698-4415-4e4b-ba53-9957262d1542"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:08:26.355724 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.355694    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-home" (OuterVolumeSpecName: "home") pod "992f4698-4415-4e4b-ba53-9957262d1542" (UID: "992f4698-4415-4e4b-ba53-9957262d1542"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:08:26.356066 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.356013    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-home\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:08:26.356066 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.356034    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-tmp-dir\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:08:26.356066 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.356048    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-model-cache\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:08:26.357736 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.357717    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992f4698-4415-4e4b-ba53-9957262d1542-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "992f4698-4415-4e4b-ba53-9957262d1542" (UID: "992f4698-4415-4e4b-ba53-9957262d1542"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 14:08:26.357860 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.357844    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-dshm" (OuterVolumeSpecName: "dshm") pod "992f4698-4415-4e4b-ba53-9957262d1542" (UID: "992f4698-4415-4e4b-ba53-9957262d1542"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:08:26.357860 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.357846    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992f4698-4415-4e4b-ba53-9957262d1542-kube-api-access-gbprt" (OuterVolumeSpecName: "kube-api-access-gbprt") pod "992f4698-4415-4e4b-ba53-9957262d1542" (UID: "992f4698-4415-4e4b-ba53-9957262d1542"). InnerVolumeSpecName "kube-api-access-gbprt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:08:26.456602 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.456567    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-dshm\") pod \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") "
Sep 15 14:08:26.456744 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.456667    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-tmp-dir\") pod \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") "
Sep 15 14:08:26.456744 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.456705    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-model-cache\") pod \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") "
Sep 15 14:08:26.456744 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.456732    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac3cc33-9acb-4215-8db9-3db1bc0be41f-tls-certs\") pod \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") "
Sep 15 14:08:26.456925 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.456764    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdmw6\" (UniqueName: \"kubernetes.io/projected/cac3cc33-9acb-4215-8db9-3db1bc0be41f-kube-api-access-zdmw6\") pod \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") "
Sep 15 14:08:26.456925 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.456820    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-home\") pod \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\" (UID: \"cac3cc33-9acb-4215-8db9-3db1bc0be41f\") "
Sep 15 14:08:26.456925 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.456862    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cac3cc33-9acb-4215-8db9-3db1bc0be41f" (UID: "cac3cc33-9acb-4215-8db9-3db1bc0be41f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:08:26.457087 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.456940    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-model-cache" (OuterVolumeSpecName: "model-cache") pod "cac3cc33-9acb-4215-8db9-3db1bc0be41f" (UID: "cac3cc33-9acb-4215-8db9-3db1bc0be41f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:08:26.457171 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.457093    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-model-cache\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:08:26.457171 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.457093    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-home" (OuterVolumeSpecName: "home") pod "cac3cc33-9acb-4215-8db9-3db1bc0be41f" (UID: "cac3cc33-9acb-4215-8db9-3db1bc0be41f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:08:26.457171 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.457110    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/992f4698-4415-4e4b-ba53-9957262d1542-tls-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:08:26.457171 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.457124    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/992f4698-4415-4e4b-ba53-9957262d1542-dshm\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:08:26.457171 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.457162    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbprt\" (UniqueName: \"kubernetes.io/projected/992f4698-4415-4e4b-ba53-9957262d1542-kube-api-access-gbprt\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:08:26.457171 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.457174    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-tmp-dir\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:08:26.458692 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.458667    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-dshm" (OuterVolumeSpecName: "dshm") pod "cac3cc33-9acb-4215-8db9-3db1bc0be41f" (UID: "cac3cc33-9acb-4215-8db9-3db1bc0be41f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:08:26.458779 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.458722    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac3cc33-9acb-4215-8db9-3db1bc0be41f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cac3cc33-9acb-4215-8db9-3db1bc0be41f" (UID: "cac3cc33-9acb-4215-8db9-3db1bc0be41f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 14:08:26.458819 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.458803    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac3cc33-9acb-4215-8db9-3db1bc0be41f-kube-api-access-zdmw6" (OuterVolumeSpecName: "kube-api-access-zdmw6") pod "cac3cc33-9acb-4215-8db9-3db1bc0be41f" (UID: "cac3cc33-9acb-4215-8db9-3db1bc0be41f"). InnerVolumeSpecName "kube-api-access-zdmw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:08:26.559052 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.558945    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac3cc33-9acb-4215-8db9-3db1bc0be41f-tls-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:08:26.559303 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.559280    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdmw6\" (UniqueName: \"kubernetes.io/projected/cac3cc33-9acb-4215-8db9-3db1bc0be41f-kube-api-access-zdmw6\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:08:26.559416 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.559403    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-home\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:08:26.559495 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:26.559486    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac3cc33-9acb-4215-8db9-3db1bc0be41f-dshm\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:08:27.320122 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:27.320018    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"
Sep 15 14:08:27.320122 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:27.320031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q" event={"ID":"cac3cc33-9acb-4215-8db9-3db1bc0be41f","Type":"ContainerDied","Data":"14396295f38b59c9e7316418175decbd5225c984f17d3aa9e81f33e201262ca3"}
Sep 15 14:08:27.320122 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:27.320100    2582 scope.go:117] "RemoveContainer" containerID="a0ade85d7d3b5b2ca85cef044bc37fd737c05e976f803b22cdc8cab91d0b84bf"
Sep 15 14:08:27.320122 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:27.320027    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"
Sep 15 14:08:27.337361 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:27.337336    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"]
Sep 15 14:08:27.341445 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:27.341421    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-66261e/premium-w4-66261e-kserve-845549df46-t2bl8"]
Sep 15 14:08:27.351726 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:27.351695    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"]
Sep 15 14:08:27.353522 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:27.353502    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-66261e/simulator-w4-66261e-kserve-6f9cbb5679-tw72q"]
Sep 15 14:08:28.646452 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:28.646411    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="992f4698-4415-4e4b-ba53-9957262d1542" path="/var/lib/kubelet/pods/992f4698-4415-4e4b-ba53-9957262d1542/volumes"
Sep 15 14:08:28.647047 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:28.647022    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cac3cc33-9acb-4215-8db9-3db1bc0be41f" path="/var/lib/kubelet/pods/cac3cc33-9acb-4215-8db9-3db1bc0be41f/volumes"
Sep 15 14:08:29.257121 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:29.257086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:30.329549 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:30.329517    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:08:30.338348 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:30.338320    2582 generic.go:356] "Generic (PLEG): container finished" podID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerID="3e9f6447ebd96af86a9ca9ef5bc77d67966d08bf6a9cb9f46390ab0b179c09fa" exitCode=0
Sep 15 14:08:30.338509 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:30.338388    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" event={"ID":"21b21433-d06d-4b57-9599-f93b2c09b1b8","Type":"ContainerDied","Data":"3e9f6447ebd96af86a9ca9ef5bc77d67966d08bf6a9cb9f46390ab0b179c09fa"}
Sep 15 14:08:30.338779 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:30.338763    2582 scope.go:117] "RemoveContainer" containerID="3e9f6447ebd96af86a9ca9ef5bc77d67966d08bf6a9cb9f46390ab0b179c09fa"
Sep 15 14:08:30.339923 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:30.339893    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 14:08:30.339997 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:30.339979    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 14:08:30.340040 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:30.340015    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 14:08:31.308648 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:31.308557    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:31.351923 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:31.351889    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" event={"ID":"21b21433-d06d-4b57-9599-f93b2c09b1b8","Type":"ContainerStarted","Data":"9bcc716de48befbd29a836c51caef6e59149f45d8fb4832c61e9e0459e47bf93"}
Sep 15 14:08:32.012631 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:32.012594    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:32.191883 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:32.191839    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:32.193156 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:32.193116    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 15 14:08:32.193283 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:32.193184    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 15 14:08:32.716810 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:32.716764    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:33.192829 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:33.192791    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 15 14:08:33.193041 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:33.192858    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 15 14:08:33.484669 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:33.484576    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:34.185659 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:34.185613    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:34.192661 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:34.192624    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 15 14:08:34.192828 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:34.192692    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 15 14:08:35.134498 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.134464    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:35.192332 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.192296    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 15 14:08:35.192740 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.192356    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 15 14:08:35.465940 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.465816    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:35.774967 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.774868    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"]
Sep 15 14:08:35.775799 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.775771    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac3cc33-9acb-4215-8db9-3db1bc0be41f" containerName="main"
Sep 15 14:08:35.775942 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.775929    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac3cc33-9acb-4215-8db9-3db1bc0be41f" containerName="main"
Sep 15 14:08:35.776032 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.776022    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="992f4698-4415-4e4b-ba53-9957262d1542" containerName="main"
Sep 15 14:08:35.776125 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.776114    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="992f4698-4415-4e4b-ba53-9957262d1542" containerName="main"
Sep 15 14:08:35.776389 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.776375    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac3cc33-9acb-4215-8db9-3db1bc0be41f" containerName="main"
Sep 15 14:08:35.776491 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.776480    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="992f4698-4415-4e4b-ba53-9957262d1542" containerName="main"
Sep 15 14:08:35.779193 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.779170    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.786161 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.784448    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d418fd-openshift-default-dockercfg-ksmn9\""
Sep 15 14:08:35.798779 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.798746    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"]
Sep 15 14:08:35.845078 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.845041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-istio-data\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.845408 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.845385    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ede612-7215-411e-93d1-625138db97d8-istiod-ca-cert\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.845544 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.845526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfzqd\" (UniqueName: \"kubernetes.io/projected/c8ede612-7215-411e-93d1-625138db97d8-kube-api-access-jfzqd\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.845663 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.845648    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-credential-socket\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.845774 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.845760    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-workload-socket\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.845883 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.845870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-istio-envoy\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.846040 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.846026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-workload-certs\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.846184 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.846168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ede612-7215-411e-93d1-625138db97d8-istio-podinfo\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.846338 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.846323    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ede612-7215-411e-93d1-625138db97d8-istio-token\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.848115 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.848077    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:35.947800 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.947764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ede612-7215-411e-93d1-625138db97d8-istio-token\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.947993 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.947810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-istio-data\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.947993 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.947831    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ede612-7215-411e-93d1-625138db97d8-istiod-ca-cert\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.947993 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.947850    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzqd\" (UniqueName: \"kubernetes.io/projected/c8ede612-7215-411e-93d1-625138db97d8-kube-api-access-jfzqd\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.948218 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.948028    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-credential-socket\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.948218 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.948105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-workload-socket\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.948218 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.948159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-istio-envoy\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.948382 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.948235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-workload-certs\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.948382 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.948270    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-istio-data\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.948382 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.948287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ede612-7215-411e-93d1-625138db97d8-istio-podinfo\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.948538 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.948379    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-credential-socket\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.948538 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.948509    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-workload-socket\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.948647 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.948580    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ede612-7215-411e-93d1-625138db97d8-istiod-ca-cert\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.948647 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.948600    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-workload-certs\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.950509 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.950488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-istio-envoy\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.950778 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.950760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ede612-7215-411e-93d1-625138db97d8-istio-podinfo\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.961180 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.961128    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ede612-7215-411e-93d1-625138db97d8-istio-token\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:35.961576 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:35.961557    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfzqd\" (UniqueName: \"kubernetes.io/projected/c8ede612-7215-411e-93d1-625138db97d8-kube-api-access-jfzqd\") pod \"e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") " pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:36.107765 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:36.107720    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:36.202173 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:36.192713    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 15 14:08:36.202173 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:36.192797    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 15 14:08:36.389549 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:36.389469    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"]
Sep 15 14:08:36.393405 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:08:36.393365    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ede612_7215_411e_93d1_625138db97d8.slice/crio-7acc30338d342b5818cdd4cffda664e78c05119f823043d83e383601ac171a39 WatchSource:0}: Error finding container 7acc30338d342b5818cdd4cffda664e78c05119f823043d83e383601ac171a39: Status 404 returned error can't find the container with id 7acc30338d342b5818cdd4cffda664e78c05119f823043d83e383601ac171a39
Sep 15 14:08:36.395894 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:36.395857    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 14:08:36.396005 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:36.395938    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 14:08:36.396005 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:36.395983    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 14:08:36.499956 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:36.499922    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:37.196925 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:37.196894    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:37.197835 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:37.197818    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:08:37.391652 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:37.391616    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" event={"ID":"c8ede612-7215-411e-93d1-625138db97d8","Type":"ContainerStarted","Data":"eb9cb3f1fc4d49d89e1bc3674042bd9d5d8c1cc09666fc29f0bd917c85a57c56"}
Sep 15 14:08:37.392016 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:37.391678    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" event={"ID":"c8ede612-7215-411e-93d1-625138db97d8","Type":"ContainerStarted","Data":"7acc30338d342b5818cdd4cffda664e78c05119f823043d83e383601ac171a39"}
Sep 15 14:08:37.415633 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:37.415582    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" podStartSLOduration=2.415565829 podStartE2EDuration="2.415565829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:08:37.412546349 +0000 UTC m=+1815.290449168" watchObservedRunningTime="2026-09-15 14:08:37.415565829 +0000 UTC m=+1815.293468695"
Sep 15 14:08:38.108869 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:38.108825    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:38.114053 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:38.114024    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:38.397101 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:38.397022    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:38.397942 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:38.397922    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:08:38.610622 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:38.610589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:39.408752 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:39.408706    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:39.505608 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:39.505577    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:39.604049 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:39.604004    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:40.317919 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:40.317881    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:40.511697 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:40.511663    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:41.580329 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:41.580298    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:42.462797 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.462766    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"]
Sep 15 14:08:42.465704 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.465682    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:42.468536 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.468352    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-d418fd-serving-cert\""
Sep 15 14:08:42.478616 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.478592    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"]
Sep 15 14:08:42.500628 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.500601    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:42.620619 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.620578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:42.621017 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.620648    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:42.621017 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.620679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j9t7\" (UniqueName: \"kubernetes.io/projected/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-kube-api-access-6j9t7\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:42.621017 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.620751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:42.722314 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.722223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:42.722314 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.722305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:42.722555 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.722334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j9t7\" (UniqueName: \"kubernetes.io/projected/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-kube-api-access-6j9t7\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:42.722555 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:08:42.722385    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-serving-cert: secret "maas-api-e2e-shared-b-w3-d418fd-serving-cert" not found
Sep 15 14:08:42.722555 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.722409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:42.722555 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:08:42.722461    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-maas-api-tls podName:e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34 nodeName:}" failed. No retries permitted until 2026-09-15 14:08:43.22243902 +0000 UTC m=+1821.100341835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-maas-api-tls") pod "maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7" (UID: "e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34") : secret "maas-api-e2e-shared-b-w3-d418fd-serving-cert" not found
Sep 15 14:08:42.723088 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.723065    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:42.724926 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.724890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:42.738871 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:42.738835    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j9t7\" (UniqueName: \"kubernetes.io/projected/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-kube-api-access-6j9t7\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:43.227293 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:43.227255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:43.227722 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:08:43.227409    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-serving-cert: secret "maas-api-e2e-shared-b-w3-d418fd-serving-cert" not found
Sep 15 14:08:43.227722 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:08:43.227634    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-maas-api-tls podName:e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34 nodeName:}" failed. No retries permitted until 2026-09-15 14:08:44.227613043 +0000 UTC m=+1822.105515859 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-maas-api-tls") pod "maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7" (UID: "e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34") : secret "maas-api-e2e-shared-b-w3-d418fd-serving-cert" not found
Sep 15 14:08:44.239235 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:44.239197    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:44.241738 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:44.241710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:44.279811 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:44.279772    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:44.553252 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:44.553226    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"]
Sep 15 14:08:44.554737 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:08:44.554703    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2eeaf7b_b4b3_4f2b_9b02_d77110b5ed34.slice/crio-27b554e6b9733a6f691e4e6d398be3016caf14cf85835371beb1f97ca0dcf2cc WatchSource:0}: Error finding container 27b554e6b9733a6f691e4e6d398be3016caf14cf85835371beb1f97ca0dcf2cc: Status 404 returned error can't find the container with id 27b554e6b9733a6f691e4e6d398be3016caf14cf85835371beb1f97ca0dcf2cc
Sep 15 14:08:45.434485 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:45.434450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7" event={"ID":"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34","Type":"ContainerStarted","Data":"27b554e6b9733a6f691e4e6d398be3016caf14cf85835371beb1f97ca0dcf2cc"}
Sep 15 14:08:45.449572 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:45.449538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:45.548648 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:45.548612    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:46.178061 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:46.178023    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:48.006980 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:48.006948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:48.103696 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:48.103666    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:48.434154 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:48.434115    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:08:48.450773 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:48.450737    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7" event={"ID":"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34","Type":"ContainerStarted","Data":"b009b2a25c5a175e697ddf2c1a50000e613f212bb33d546f77338839ef6bee1b"}
Sep 15 14:08:48.450911 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:48.450856    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:48.476866 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:48.476812    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7" podStartSLOduration=3.088174545 podStartE2EDuration="6.476797307s" totalImagesPullingTime="3.388622762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:08:48.474408954 +0000 UTC m=+1826.352311773" watchObservedRunningTime="2026-09-15 14:08:48.476797307 +0000 UTC m=+1826.354700126"
Sep 15 14:08:48.809895 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:48.809808    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:54.461445 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:54.461408    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:08:54.651042 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:54.651010    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:08:59.530451 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:59.530414    2582 generic.go:356] "Generic (PLEG): container finished" podID="c8ede612-7215-411e-93d1-625138db97d8" containerID="eb9cb3f1fc4d49d89e1bc3674042bd9d5d8c1cc09666fc29f0bd917c85a57c56" exitCode=0
Sep 15 14:08:59.530893 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:59.530482    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" event={"ID":"c8ede612-7215-411e-93d1-625138db97d8","Type":"ContainerDied","Data":"eb9cb3f1fc4d49d89e1bc3674042bd9d5d8c1cc09666fc29f0bd917c85a57c56"}
Sep 15 14:08:59.530893 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:59.530853    2582 scope.go:117] "RemoveContainer" containerID="eb9cb3f1fc4d49d89e1bc3674042bd9d5d8c1cc09666fc29f0bd917c85a57c56"
Sep 15 14:08:59.532014 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:59.531983    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 14:08:59.532123 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:59.532081    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 14:08:59.532218 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:08:59.532131    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 14:09:00.539104 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:00.539065    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" event={"ID":"c8ede612-7215-411e-93d1-625138db97d8","Type":"ContainerStarted","Data":"8c4b2687164d573a67b24a4fa617cd90a6501efca2fde2e48523018a08c10faf"}
Sep 15 14:09:00.772167 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:00.768675    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:01.108608 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:01.108569    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:09:01.108797 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:01.108753    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 15 14:09:01.108850 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:01.108801    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" podUID="c8ede612-7215-411e-93d1-625138db97d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 15 14:09:01.211654 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:01.211625    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:02.011746 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:02.011712    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:02.108169 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:02.108103    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 15 14:09:02.108358 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:02.108194    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" podUID="c8ede612-7215-411e-93d1-625138db97d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 15 14:09:02.702123 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:02.702079    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:03.108332 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:03.108280    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 15 14:09:03.108806 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:03.108358    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" podUID="c8ede612-7215-411e-93d1-625138db97d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 15 14:09:03.508519 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:03.508400    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:04.108606 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:04.108570    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 15 14:09:04.108990 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:04.108627    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" podUID="c8ede612-7215-411e-93d1-625138db97d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 15 14:09:04.223495 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:04.223460    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:04.381416 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:04.381319    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:05.108757 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:05.108721    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 15 14:09:05.109172 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:05.108780    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" podUID="c8ede612-7215-411e-93d1-625138db97d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 15 14:09:05.210880 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:05.210835    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:05.915629 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:05.915596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:06.112722 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:06.112687    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:09:06.113171 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:06.113121    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:09:06.113714 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:06.113698    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:09:06.804325 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:06.804280    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:09.587448 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:09.587415    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:09.659427 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:09.659394    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:12.221834 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:12.221798    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:09:18.486201 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:18.486166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:18.544800 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:18.544769    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:22.802067 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:22.802036    2582 scope.go:117] "RemoveContainer" containerID="db8798987a21d3d3e6d2eea22054a541e8fb91153f0b8c42fc4c9860a9cf9af2"
Sep 15 14:09:24.132230 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.132193    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"]
Sep 15 14:09:24.135813 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.135789    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.140529 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.140304    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a090a0\"/\"auth-test-model-a090a0-kserve-self-signed-certs\""
Sep 15 14:09:24.140529 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.140371    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a090a0\"/\"kube-root-ca.crt\""
Sep 15 14:09:24.140529 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.140379    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a090a0\"/\"openshift-service-ca.crt\""
Sep 15 14:09:24.140529 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.140508    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a090a0\"/\"default-dockercfg-zfbxt\""
Sep 15 14:09:24.143741 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.143716    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"]
Sep 15 14:09:24.167951 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.167916    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:24.222620 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.222585    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-dshm\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.222813 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.222651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-tmp-dir\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.222813 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.222692    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-home\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.222813 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.222706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95e2950-8361-4453-9464-1c26a365c835-tls-certs\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.222813 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.222724    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-model-cache\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.222985 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.222825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47txd\" (UniqueName: \"kubernetes.io/projected/d95e2950-8361-4453-9464-1c26a365c835-kube-api-access-47txd\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.324196 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.324129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-dshm\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.324401 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.324243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-tmp-dir\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.324401 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.324284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-home\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.324401 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.324304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95e2950-8361-4453-9464-1c26a365c835-tls-certs\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.324401 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.324331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-model-cache\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.324401 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.324383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47txd\" (UniqueName: \"kubernetes.io/projected/d95e2950-8361-4453-9464-1c26a365c835-kube-api-access-47txd\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.324692 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.324662    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-tmp-dir\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.324754 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.324673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-home\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.324754 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.324735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-model-cache\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.326594 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.326566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-dshm\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.326882 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.326862    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95e2950-8361-4453-9464-1c26a365c835-tls-certs\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.333043 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.333024    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47txd\" (UniqueName: \"kubernetes.io/projected/d95e2950-8361-4453-9464-1c26a365c835-kube-api-access-47txd\") pod \"auth-test-model-a090a0-kserve-584596446c-wllsk\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") " pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.447551 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.447459    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:24.718126 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:24.718099    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"]
Sep 15 14:09:24.720014 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:09:24.719981    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd95e2950_8361_4453_9464_1c26a365c835.slice/crio-5bbb82669d4e27c920212c984cbf2ea7227cf535cf3ab0899e86792a750fe5ce WatchSource:0}: Error finding container 5bbb82669d4e27c920212c984cbf2ea7227cf535cf3ab0899e86792a750fe5ce: Status 404 returned error can't find the container with id 5bbb82669d4e27c920212c984cbf2ea7227cf535cf3ab0899e86792a750fe5ce
Sep 15 14:09:25.671392 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:25.671351    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk" event={"ID":"d95e2950-8361-4453-9464-1c26a365c835","Type":"ContainerStarted","Data":"5c6471caa0374751e060409b5c437dafbec120817b3748f3773eb6092548070c"}
Sep 15 14:09:25.671771 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:25.671404    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk" event={"ID":"d95e2950-8361-4453-9464-1c26a365c835","Type":"ContainerStarted","Data":"5bbb82669d4e27c920212c984cbf2ea7227cf535cf3ab0899e86792a750fe5ce"}
Sep 15 14:09:25.693179 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:25.693115    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk" podStartSLOduration=1.693099486 podStartE2EDuration="1.693099486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:09:25.691314554 +0000 UTC m=+1863.569217376" watchObservedRunningTime="2026-09-15 14:09:25.693099486 +0000 UTC m=+1863.571002339"
Sep 15 14:09:29.524750 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:29.524718    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:09:34.447765 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:34.447673    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:34.448289 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:34.447787    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:34.460383 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:34.460356    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:34.734045 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:34.733954    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:09:43.224094 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.224058    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"]
Sep 15 14:09:43.227928 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.227906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.230632 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.230608    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d418fd\"/\"openshift-service-ca.crt\""
Sep 15 14:09:43.230632 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.230629    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d418fd\"/\"kube-root-ca.crt\""
Sep 15 14:09:43.231441 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.231425    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d418fd\"/\"default-dockercfg-64kmd\""
Sep 15 14:09:43.231503 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.231466    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d418fd\"/\"auth-test-model-d418fd-kserve-self-signed-certs\""
Sep 15 14:09:43.239353 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.239329    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"]
Sep 15 14:09:43.301111 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.301064    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-home\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.301288 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.301131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-tmp-dir\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.301288 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.301243    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-tls-certs\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.301403 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.301291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-dshm\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.301403 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.301363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wcx8\" (UniqueName: \"kubernetes.io/projected/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-kube-api-access-8wcx8\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.301522 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.301408    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-model-cache\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.341355 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.341326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:09:43.402908 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.402870    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-home\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.403081 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.402916    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-tmp-dir\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.403124 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.403088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-tls-certs\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.403124 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.403117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-dshm\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.403238 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.403181    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wcx8\" (UniqueName: \"kubernetes.io/projected/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-kube-api-access-8wcx8\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.403238 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.403219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-model-cache\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.403377 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.403353    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-home\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.403421 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.403390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-tmp-dir\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.403643 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.403616    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-model-cache\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.405333 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.405309    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-dshm\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.405585 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.405565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-tls-certs\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.411295 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.411271    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wcx8\" (UniqueName: \"kubernetes.io/projected/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-kube-api-access-8wcx8\") pod \"auth-test-model-d418fd-kserve-b4cd5c447-jfj4n\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") " pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.538528 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.538439    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:43.823654 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:43.823618    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"]
Sep 15 14:09:43.827491 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:09:43.827447    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16edf5d7_b4e1_492f_97e7_48b34ad75cd7.slice/crio-06394e3a9ea3c51300b33bd63ab0d7f5f245f0c426c0b567a36b7e1fe96bb56a WatchSource:0}: Error finding container 06394e3a9ea3c51300b33bd63ab0d7f5f245f0c426c0b567a36b7e1fe96bb56a: Status 404 returned error can't find the container with id 06394e3a9ea3c51300b33bd63ab0d7f5f245f0c426c0b567a36b7e1fe96bb56a
Sep 15 14:09:44.773972 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:44.773878    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n" event={"ID":"16edf5d7-b4e1-492f-97e7-48b34ad75cd7","Type":"ContainerStarted","Data":"185314415a3726b339e9b0a3c760db989297edb243212923c77469b096c232c0"}
Sep 15 14:09:44.773972 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:44.773932    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n" event={"ID":"16edf5d7-b4e1-492f-97e7-48b34ad75cd7","Type":"ContainerStarted","Data":"06394e3a9ea3c51300b33bd63ab0d7f5f245f0c426c0b567a36b7e1fe96bb56a"}
Sep 15 14:09:44.793241 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:44.793182    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n" podStartSLOduration=1.7931664 podStartE2EDuration="1.7931664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:09:44.789916161 +0000 UTC m=+1882.667818981" watchObservedRunningTime="2026-09-15 14:09:44.7931664 +0000 UTC m=+1882.671069218"
Sep 15 14:09:48.930947 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:48.930912    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:09:53.539188 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:53.539127    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:53.539573 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:53.539202    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:53.551559 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:53.551526    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:09:53.836542 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:09:53.836518    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:10:03.025238 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:10:03.025196    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:10:03.092543 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:10:03.092507    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:10:10.554612 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:10:10.554579    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:10:10.770654 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:10:10.770619    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:10:16.032990 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:10:16.032947    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:10:29.244510 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:10:29.244473    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:10:29.371071 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:10:29.371034    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:10:29.474524 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:10:29.474490    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:11:27.342763 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:11:27.342683    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:11:36.418290 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:11:36.418244    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:11:42.235942 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:11:42.235909    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:11:51.788503 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:11:51.788466    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:11:51.889780 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:11:51.889747    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:11:53.957674 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:11:53.957639    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:12:08.614978 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:12:08.614937    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:12:33.118364 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:12:33.118328    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:12:39.021136 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:12:39.021095    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:12:39.087715 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:12:39.087678    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:12:39.601290 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:12:39.601258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:12:39.696922 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:12:39.696893    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:12:50.163516 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:12:50.163421    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:12:50.390186 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:12:50.390128    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:06.699029 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:06.698995    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:06.914150 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:06.914111    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:11.265841 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:11.265801    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:11.930872 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:11.930836    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:13:13.258519 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:13.258481    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:17.408955 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:17.408921    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:17.507739 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:17.507703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:17.747388 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:17.747311    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:18.058677 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:18.058580    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:18.861721 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:18.861691    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:20.898002 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:20.897961    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:20.991121 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:20.991094    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:21.095710 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:21.095680    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:21.949714 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:21.949673    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:22.817967 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:22.817931    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:23.664315 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:23.664271    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:24.685616 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:24.685581    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:25.499503 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:25.499443    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:25.599216 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:25.599178    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:25.698961 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:25.698925    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:25.956671 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:25.956640    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:26.029708 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:26.029671    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:26.465123 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:26.465092    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:27.402232 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:27.402185    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:30.065102 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:30.065065    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:31.173753 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:31.173719    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:31.593797 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:31.593769    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:32.537063 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:32.537027    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:13:34.781703 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:34.781659    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:34.987004 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:34.986972    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:35.765787 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:35.765710    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:36.635331 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:36.635296    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:37.372610 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:37.372573    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:38.094934 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:38.094884    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:38.988344 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:38.988300    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:39.693732 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:39.693685    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:40.387012 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:40.386976    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:40.804155 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:40.804057    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:41.218395 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:41.218363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:41.367364 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:41.367326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:41.526083 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:41.525989    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:41.795603 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:41.795511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:44.401283 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:44.401236    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:45.031950 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:45.031917    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:45.137641 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:45.137600    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:45.291819 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:45.291740    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:46.103002 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:46.102969    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:46.895604 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:46.895573    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:47.652609 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:47.652577    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:48.396195 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:48.396123    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:49.195745 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:49.195713    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:49.297001 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:49.296968    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:49.379900 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:49.379868    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:49.765614 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:49.765581    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:49.865010 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:49.864971    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:49.896238 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:49.896201    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"]
Sep 15 14:13:49.899993 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:49.899960    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"
Sep 15 14:13:49.902702 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:49.902672    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e50cdb-dockercfg-xss6t\""
Sep 15 14:13:49.903266 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:49.903248    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e50cdb\""
Sep 15 14:13:49.911916 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:49.911886    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"]
Sep 15 14:13:50.004362 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:50.004317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d7412d-3cc0-4299-89ec-0f1aec3c782a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e50cdb-799789b575-2xm25\" (UID: \"97d7412d-3cc0-4299-89ec-0f1aec3c782a\") " pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"
Sep 15 14:13:50.004625 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:50.004599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q68rc\" (UniqueName: \"kubernetes.io/projected/97d7412d-3cc0-4299-89ec-0f1aec3c782a-kube-api-access-q68rc\") pod \"payload-processing-e2e-api-b-e50cdb-799789b575-2xm25\" (UID: \"97d7412d-3cc0-4299-89ec-0f1aec3c782a\") " pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"
Sep 15 14:13:50.106198 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:50.106159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q68rc\" (UniqueName: \"kubernetes.io/projected/97d7412d-3cc0-4299-89ec-0f1aec3c782a-kube-api-access-q68rc\") pod \"payload-processing-e2e-api-b-e50cdb-799789b575-2xm25\" (UID: \"97d7412d-3cc0-4299-89ec-0f1aec3c782a\") " pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"
Sep 15 14:13:50.106421 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:50.106265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d7412d-3cc0-4299-89ec-0f1aec3c782a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e50cdb-799789b575-2xm25\" (UID: \"97d7412d-3cc0-4299-89ec-0f1aec3c782a\") " pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"
Sep 15 14:13:50.107116 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:50.107091    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d7412d-3cc0-4299-89ec-0f1aec3c782a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e50cdb-799789b575-2xm25\" (UID: \"97d7412d-3cc0-4299-89ec-0f1aec3c782a\") " pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"
Sep 15 14:13:50.114848 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:50.114818    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q68rc\" (UniqueName: \"kubernetes.io/projected/97d7412d-3cc0-4299-89ec-0f1aec3c782a-kube-api-access-q68rc\") pod \"payload-processing-e2e-api-b-e50cdb-799789b575-2xm25\" (UID: \"97d7412d-3cc0-4299-89ec-0f1aec3c782a\") " pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"
Sep 15 14:13:50.212809 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:50.212766    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"
Sep 15 14:13:50.350871 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:50.350837    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:50.492009 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:50.491980    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"]
Sep 15 14:13:51.047087 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:51.047048    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25" event={"ID":"97d7412d-3cc0-4299-89ec-0f1aec3c782a","Type":"ContainerStarted","Data":"c5e29f668f4bc5787629be6ac39cccfa4935809f31a0764c92018c885dc23fe0"}
Sep 15 14:13:51.047293 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:51.047105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25" event={"ID":"97d7412d-3cc0-4299-89ec-0f1aec3c782a","Type":"ContainerStarted","Data":"7d11554ffae4ffd01a6064674fc810cf2fbbea80d4f8e19a6cc0745cf5a18ce1"}
Sep 15 14:13:51.047293 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:51.047193    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"
Sep 15 14:13:51.058318 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:51.058290    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:51.068576 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:51.068513    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25" podStartSLOduration=2.068492983 podStartE2EDuration="2.068492983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:13:51.068128843 +0000 UTC m=+2128.946031663" watchObservedRunningTime="2026-09-15 14:13:51.068492983 +0000 UTC m=+2128.946395803"
Sep 15 14:13:52.024642 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:52.024607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:13:56.425044 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:13:56.425005    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:14:02.054249 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:02.054215    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"
Sep 15 14:14:06.766101 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:06.766066    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:10.321284 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:10.321248    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:11.195654 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:11.195621    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:11.297652 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:11.297605    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:11.720684 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:11.720648    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:12.465456 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:12.465414    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:13.479216 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:13.479185    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:14.306985 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:14.306952    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:15.083229 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:15.083131    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:15.184179 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:15.184134    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:16.014433 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:16.014402    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:16.722607 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:16.722567    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:18.283628 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:18.283598    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:19.121841 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:19.121796    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:20.003915 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:20.003878    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:20.696492 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:20.696453    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:21.477526 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:21.477487    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:22.248571 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:22.248532    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:23.132540 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:23.132503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:24.020405 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:24.020373    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:33.537320 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:33.537282    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:34.899684 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:34.899640    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:35.006972 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:35.006931    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:35.320627 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:35.320532    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:36.102365 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:36.102336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:36.841018 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:36.840983    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:37.705344 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:37.705308    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:38.422235 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:38.422201    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:39.122403 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:39.122369    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:14:40.446871 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:40.446836    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:40.547406 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:40.547367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:40.695425 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:40.695396    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:42.000779 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:42.000743    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:42.733872 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:42.733841    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:43.699614 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:43.699581    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:44.443568 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:44.443524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:45.229096 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:45.229055    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:46.102350 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:46.102314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:56.700289 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:56.700246    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:56.749297 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:56.749259    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"]
Sep 15 14:14:56.749590 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:56.749539    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25" podUID="97d7412d-3cc0-4299-89ec-0f1aec3c782a" containerName="payload-processing" containerID="cri-o://c5e29f668f4bc5787629be6ac39cccfa4935809f31a0764c92018c885dc23fe0" gracePeriod=30
Sep 15 14:14:56.791159 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:56.791114    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:56.897906 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:56.897875    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:57.001702 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.001673    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:57.016201 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.016179    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"
Sep 15 14:14:57.079788 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.079758    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:57.157134 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.157096    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q68rc\" (UniqueName: \"kubernetes.io/projected/97d7412d-3cc0-4299-89ec-0f1aec3c782a-kube-api-access-q68rc\") pod \"97d7412d-3cc0-4299-89ec-0f1aec3c782a\" (UID: \"97d7412d-3cc0-4299-89ec-0f1aec3c782a\") "
Sep 15 14:14:57.157313 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.157214    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d7412d-3cc0-4299-89ec-0f1aec3c782a-plugins-config-volume\") pod \"97d7412d-3cc0-4299-89ec-0f1aec3c782a\" (UID: \"97d7412d-3cc0-4299-89ec-0f1aec3c782a\") "
Sep 15 14:14:57.157583 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.157561    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d7412d-3cc0-4299-89ec-0f1aec3c782a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "97d7412d-3cc0-4299-89ec-0f1aec3c782a" (UID: "97d7412d-3cc0-4299-89ec-0f1aec3c782a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 14:14:57.159389 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.159367    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d7412d-3cc0-4299-89ec-0f1aec3c782a-kube-api-access-q68rc" (OuterVolumeSpecName: "kube-api-access-q68rc") pod "97d7412d-3cc0-4299-89ec-0f1aec3c782a" (UID: "97d7412d-3cc0-4299-89ec-0f1aec3c782a"). InnerVolumeSpecName "kube-api-access-q68rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:14:57.258898 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.258817    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q68rc\" (UniqueName: \"kubernetes.io/projected/97d7412d-3cc0-4299-89ec-0f1aec3c782a-kube-api-access-q68rc\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:14:57.258898 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.258845    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d7412d-3cc0-4299-89ec-0f1aec3c782a-plugins-config-volume\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:14:57.336698 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.336662    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:57.411739 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.411703    2582 generic.go:356] "Generic (PLEG): container finished" podID="97d7412d-3cc0-4299-89ec-0f1aec3c782a" containerID="c5e29f668f4bc5787629be6ac39cccfa4935809f31a0764c92018c885dc23fe0" exitCode=0
Sep 15 14:14:57.411914 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.411753    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"
Sep 15 14:14:57.411914 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.411793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25" event={"ID":"97d7412d-3cc0-4299-89ec-0f1aec3c782a","Type":"ContainerDied","Data":"c5e29f668f4bc5787629be6ac39cccfa4935809f31a0764c92018c885dc23fe0"}
Sep 15 14:14:57.411914 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.411853    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25" event={"ID":"97d7412d-3cc0-4299-89ec-0f1aec3c782a","Type":"ContainerDied","Data":"7d11554ffae4ffd01a6064674fc810cf2fbbea80d4f8e19a6cc0745cf5a18ce1"}
Sep 15 14:14:57.411914 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.411881    2582 scope.go:117] "RemoveContainer" containerID="c5e29f668f4bc5787629be6ac39cccfa4935809f31a0764c92018c885dc23fe0"
Sep 15 14:14:57.421414 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.421395    2582 scope.go:117] "RemoveContainer" containerID="c5e29f668f4bc5787629be6ac39cccfa4935809f31a0764c92018c885dc23fe0"
Sep 15 14:14:57.421710 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:14:57.421688    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e29f668f4bc5787629be6ac39cccfa4935809f31a0764c92018c885dc23fe0\": container with ID starting with c5e29f668f4bc5787629be6ac39cccfa4935809f31a0764c92018c885dc23fe0 not found: ID does not exist" containerID="c5e29f668f4bc5787629be6ac39cccfa4935809f31a0764c92018c885dc23fe0"
Sep 15 14:14:57.421787 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.421719    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e29f668f4bc5787629be6ac39cccfa4935809f31a0764c92018c885dc23fe0"} err="failed to get container status \"c5e29f668f4bc5787629be6ac39cccfa4935809f31a0764c92018c885dc23fe0\": rpc error: code = NotFound desc = could not find container \"c5e29f668f4bc5787629be6ac39cccfa4935809f31a0764c92018c885dc23fe0\": container with ID starting with c5e29f668f4bc5787629be6ac39cccfa4935809f31a0764c92018c885dc23fe0 not found: ID does not exist"
Sep 15 14:14:57.441213 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.441186    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"]
Sep 15 14:14:57.445429 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.445397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:57.450294 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:57.450274    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e50cdb-799789b575-2xm25"]
Sep 15 14:14:58.232544 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:58.232513    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:58.643922 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:58.643885    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97d7412d-3cc0-4299-89ec-0f1aec3c782a" path="/var/lib/kubelet/pods/97d7412d-3cc0-4299-89ec-0f1aec3c782a/volumes"
Sep 15 14:14:58.998865 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:58.998766    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:14:59.794828 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:14:59.794793    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:00.264017 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.263980    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5"]
Sep 15 14:15:00.264438 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.264422    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97d7412d-3cc0-4299-89ec-0f1aec3c782a" containerName="payload-processing"
Sep 15 14:15:00.264438 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.264438    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="97d7412d-3cc0-4299-89ec-0f1aec3c782a" containerName="payload-processing"
Sep 15 14:15:00.264577 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.264501    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="97d7412d-3cc0-4299-89ec-0f1aec3c782a" containerName="payload-processing"
Sep 15 14:15:00.267735 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.267705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5"
Sep 15 14:15:00.269003 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.268976    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j"]
Sep 15 14:15:00.270312 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.270290    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzmkv\""
Sep 15 14:15:00.273001 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.272985    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j"
Sep 15 14:15:00.282650 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.282621    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5"]
Sep 15 14:15:00.285993 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.285959    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j"]
Sep 15 14:15:00.392943 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.392911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxm7f\" (UniqueName: \"kubernetes.io/projected/ad343043-b04a-4ea6-958e-fe45cb33503f-kube-api-access-jxm7f\") pod \"maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j\" (UID: \"ad343043-b04a-4ea6-958e-fe45cb33503f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j"
Sep 15 14:15:00.393118 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.392975    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8v2f\" (UniqueName: \"kubernetes.io/projected/1078ca19-bc0d-4030-8d57-84c5bac3ed4c-kube-api-access-p8v2f\") pod \"maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5\" (UID: \"1078ca19-bc0d-4030-8d57-84c5bac3ed4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5"
Sep 15 14:15:00.493666 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.493621    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxm7f\" (UniqueName: \"kubernetes.io/projected/ad343043-b04a-4ea6-958e-fe45cb33503f-kube-api-access-jxm7f\") pod \"maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j\" (UID: \"ad343043-b04a-4ea6-958e-fe45cb33503f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j"
Sep 15 14:15:00.493852 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.493698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8v2f\" (UniqueName: \"kubernetes.io/projected/1078ca19-bc0d-4030-8d57-84c5bac3ed4c-kube-api-access-p8v2f\") pod \"maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5\" (UID: \"1078ca19-bc0d-4030-8d57-84c5bac3ed4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5"
Sep 15 14:15:00.502979 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.502948    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxm7f\" (UniqueName: \"kubernetes.io/projected/ad343043-b04a-4ea6-958e-fe45cb33503f-kube-api-access-jxm7f\") pod \"maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j\" (UID: \"ad343043-b04a-4ea6-958e-fe45cb33503f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j"
Sep 15 14:15:00.503104 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.503054    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8v2f\" (UniqueName: \"kubernetes.io/projected/1078ca19-bc0d-4030-8d57-84c5bac3ed4c-kube-api-access-p8v2f\") pod \"maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5\" (UID: \"1078ca19-bc0d-4030-8d57-84c5bac3ed4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5"
Sep 15 14:15:00.581953 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.581874    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5"
Sep 15 14:15:00.593641 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.593618    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j"
Sep 15 14:15:00.740253 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.740218    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:00.856271 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.856240    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5"]
Sep 15 14:15:00.857919 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:15:00.857892    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1078ca19_bc0d_4030_8d57_84c5bac3ed4c.slice/crio-8edef52308eb045a84612b10680e4ad047b12de75691b591a0767d9554cd63e7 WatchSource:0}: Error finding container 8edef52308eb045a84612b10680e4ad047b12de75691b591a0767d9554cd63e7: Status 404 returned error can't find the container with id 8edef52308eb045a84612b10680e4ad047b12de75691b591a0767d9554cd63e7
Sep 15 14:15:00.874126 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:00.874102    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j"]
Sep 15 14:15:00.875834 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:15:00.875810    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad343043_b04a_4ea6_958e_fe45cb33503f.slice/crio-34a8ced3ac1227495eeae47b2bf7f3c87b4f0a7240ea94ba2c2d396b53a568e6 WatchSource:0}: Error finding container 34a8ced3ac1227495eeae47b2bf7f3c87b4f0a7240ea94ba2c2d396b53a568e6: Status 404 returned error can't find the container with id 34a8ced3ac1227495eeae47b2bf7f3c87b4f0a7240ea94ba2c2d396b53a568e6
Sep 15 14:15:01.438527 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:01.438416    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5" event={"ID":"1078ca19-bc0d-4030-8d57-84c5bac3ed4c","Type":"ContainerStarted","Data":"69b923694c306856e9d1f251ff16e62bc5d169cc313532349c64c8a341989c79"}
Sep 15 14:15:01.438527 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:01.438480    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5" event={"ID":"1078ca19-bc0d-4030-8d57-84c5bac3ed4c","Type":"ContainerStarted","Data":"8edef52308eb045a84612b10680e4ad047b12de75691b591a0767d9554cd63e7"}
Sep 15 14:15:01.439893 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:01.439872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j" event={"ID":"ad343043-b04a-4ea6-958e-fe45cb33503f","Type":"ContainerStarted","Data":"058accbbf4c093fecd12dcd621e0cec5f1278afe56e0c14264c3d6215da3f192"}
Sep 15 14:15:01.440016 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:01.439907    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j" event={"ID":"ad343043-b04a-4ea6-958e-fe45cb33503f","Type":"ContainerStarted","Data":"34a8ced3ac1227495eeae47b2bf7f3c87b4f0a7240ea94ba2c2d396b53a568e6"}
Sep 15 14:15:01.456882 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:01.456829    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5" podStartSLOduration=1.456813648 podStartE2EDuration="1.456813648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:15:01.455047718 +0000 UTC m=+2199.332950537" watchObservedRunningTime="2026-09-15 14:15:01.456813648 +0000 UTC m=+2199.334716466"
Sep 15 14:15:01.473184 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:01.473116    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j" podStartSLOduration=1.473100086 podStartE2EDuration="1.473100086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:15:01.470023183 +0000 UTC m=+2199.347926004" watchObservedRunningTime="2026-09-15 14:15:01.473100086 +0000 UTC m=+2199.351002951"
Sep 15 14:15:02.124866 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:02.124831    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:15:02.447730 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:02.447692    2582 generic.go:356] "Generic (PLEG): container finished" podID="1078ca19-bc0d-4030-8d57-84c5bac3ed4c" containerID="69b923694c306856e9d1f251ff16e62bc5d169cc313532349c64c8a341989c79" exitCode=0
Sep 15 14:15:02.448002 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:02.447779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5" event={"ID":"1078ca19-bc0d-4030-8d57-84c5bac3ed4c","Type":"ContainerDied","Data":"69b923694c306856e9d1f251ff16e62bc5d169cc313532349c64c8a341989c79"}
Sep 15 14:15:02.449314 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:02.449290    2582 generic.go:356] "Generic (PLEG): container finished" podID="ad343043-b04a-4ea6-958e-fe45cb33503f" containerID="058accbbf4c093fecd12dcd621e0cec5f1278afe56e0c14264c3d6215da3f192" exitCode=0
Sep 15 14:15:02.449418 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:02.449315    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j" event={"ID":"ad343043-b04a-4ea6-958e-fe45cb33503f","Type":"ContainerDied","Data":"058accbbf4c093fecd12dcd621e0cec5f1278afe56e0c14264c3d6215da3f192"}
Sep 15 14:15:03.342746 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.342703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:03.504047 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.504009    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:03.600922 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.600841    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j"
Sep 15 14:15:03.622131 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.622069    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxm7f\" (UniqueName: \"kubernetes.io/projected/ad343043-b04a-4ea6-958e-fe45cb33503f-kube-api-access-jxm7f\") pod \"ad343043-b04a-4ea6-958e-fe45cb33503f\" (UID: \"ad343043-b04a-4ea6-958e-fe45cb33503f\") "
Sep 15 14:15:03.624748 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.624721    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad343043-b04a-4ea6-958e-fe45cb33503f-kube-api-access-jxm7f" (OuterVolumeSpecName: "kube-api-access-jxm7f") pod "ad343043-b04a-4ea6-958e-fe45cb33503f" (UID: "ad343043-b04a-4ea6-958e-fe45cb33503f"). InnerVolumeSpecName "kube-api-access-jxm7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:15:03.643807 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.643780    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5"
Sep 15 14:15:03.723362 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.723329    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8v2f\" (UniqueName: \"kubernetes.io/projected/1078ca19-bc0d-4030-8d57-84c5bac3ed4c-kube-api-access-p8v2f\") pod \"1078ca19-bc0d-4030-8d57-84c5bac3ed4c\" (UID: \"1078ca19-bc0d-4030-8d57-84c5bac3ed4c\") "
Sep 15 14:15:03.723566 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.723554    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxm7f\" (UniqueName: \"kubernetes.io/projected/ad343043-b04a-4ea6-958e-fe45cb33503f-kube-api-access-jxm7f\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:15:03.725500 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.725474    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1078ca19-bc0d-4030-8d57-84c5bac3ed4c-kube-api-access-p8v2f" (OuterVolumeSpecName: "kube-api-access-p8v2f") pod "1078ca19-bc0d-4030-8d57-84c5bac3ed4c" (UID: "1078ca19-bc0d-4030-8d57-84c5bac3ed4c"). InnerVolumeSpecName "kube-api-access-p8v2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:15:03.824906 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.824864    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8v2f\" (UniqueName: \"kubernetes.io/projected/1078ca19-bc0d-4030-8d57-84c5bac3ed4c-kube-api-access-p8v2f\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:15:03.950504 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.950420    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:03.958907 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.958877    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"]
Sep 15 14:15:03.959306 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.959294    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1078ca19-bc0d-4030-8d57-84c5bac3ed4c" containerName="cleanup"
Sep 15 14:15:03.959354 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.959308    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1078ca19-bc0d-4030-8d57-84c5bac3ed4c" containerName="cleanup"
Sep 15 14:15:03.959354 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.959333    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad343043-b04a-4ea6-958e-fe45cb33503f" containerName="cleanup"
Sep 15 14:15:03.959354 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.959339    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad343043-b04a-4ea6-958e-fe45cb33503f" containerName="cleanup"
Sep 15 14:15:03.959455 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.959398    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad343043-b04a-4ea6-958e-fe45cb33503f" containerName="cleanup"
Sep 15 14:15:03.959455 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.959411    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1078ca19-bc0d-4030-8d57-84c5bac3ed4c" containerName="cleanup"
Sep 15 14:15:03.962541 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.962523    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"
Sep 15 14:15:03.976113 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:03.976079    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"]
Sep 15 14:15:04.026789 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.026758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8079543-ca0a-44eb-be12-1454fc54a380-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx\" (UID: \"d8079543-ca0a-44eb-be12-1454fc54a380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"
Sep 15 14:15:04.026971 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.026806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhbx9\" (UniqueName: \"kubernetes.io/projected/d8079543-ca0a-44eb-be12-1454fc54a380-kube-api-access-hhbx9\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx\" (UID: \"d8079543-ca0a-44eb-be12-1454fc54a380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"
Sep 15 14:15:04.128248 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.128191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8079543-ca0a-44eb-be12-1454fc54a380-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx\" (UID: \"d8079543-ca0a-44eb-be12-1454fc54a380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"
Sep 15 14:15:04.128466 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.128272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhbx9\" (UniqueName: \"kubernetes.io/projected/d8079543-ca0a-44eb-be12-1454fc54a380-kube-api-access-hhbx9\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx\" (UID: \"d8079543-ca0a-44eb-be12-1454fc54a380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"
Sep 15 14:15:04.128819 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.128797    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8079543-ca0a-44eb-be12-1454fc54a380-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx\" (UID: \"d8079543-ca0a-44eb-be12-1454fc54a380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"
Sep 15 14:15:04.136658 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.136633    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhbx9\" (UniqueName: \"kubernetes.io/projected/d8079543-ca0a-44eb-be12-1454fc54a380-kube-api-access-hhbx9\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx\" (UID: \"d8079543-ca0a-44eb-be12-1454fc54a380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"
Sep 15 14:15:04.277421 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.277324    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"
Sep 15 14:15:04.283452 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.283419    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:04.461941 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.461913    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j"
Sep 15 14:15:04.462434 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.461946    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j" event={"ID":"ad343043-b04a-4ea6-958e-fe45cb33503f","Type":"ContainerDied","Data":"34a8ced3ac1227495eeae47b2bf7f3c87b4f0a7240ea94ba2c2d396b53a568e6"}
Sep 15 14:15:04.462434 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.462022    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34a8ced3ac1227495eeae47b2bf7f3c87b4f0a7240ea94ba2c2d396b53a568e6"
Sep 15 14:15:04.463399 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.463362    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5" event={"ID":"1078ca19-bc0d-4030-8d57-84c5bac3ed4c","Type":"ContainerDied","Data":"8edef52308eb045a84612b10680e4ad047b12de75691b591a0767d9554cd63e7"}
Sep 15 14:15:04.463522 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.463407    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8edef52308eb045a84612b10680e4ad047b12de75691b591a0767d9554cd63e7"
Sep 15 14:15:04.463522 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.463374    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5"
Sep 15 14:15:04.550232 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:04.550205    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"]
Sep 15 14:15:04.551890 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:15:04.551864    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8079543_ca0a_44eb_be12_1454fc54a380.slice/crio-e520f0e9459b1c359bac4ce03f4dd0e30e93e489930f8a5580a7eaf321c00ce8 WatchSource:0}: Error finding container e520f0e9459b1c359bac4ce03f4dd0e30e93e489930f8a5580a7eaf321c00ce8: Status 404 returned error can't find the container with id e520f0e9459b1c359bac4ce03f4dd0e30e93e489930f8a5580a7eaf321c00ce8
Sep 15 14:15:05.080236 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:05.080206    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:05.472643 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:05.472609    2582 generic.go:356] "Generic (PLEG): container finished" podID="d8079543-ca0a-44eb-be12-1454fc54a380" containerID="2429e8a20819bd114cdbbed70ee6b3f078d65b8cdc1b42d470ab148be5bc5cda" exitCode=0
Sep 15 14:15:05.473023 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:05.472698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx" event={"ID":"d8079543-ca0a-44eb-be12-1454fc54a380","Type":"ContainerDied","Data":"2429e8a20819bd114cdbbed70ee6b3f078d65b8cdc1b42d470ab148be5bc5cda"}
Sep 15 14:15:05.473023 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:05.472752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx" event={"ID":"d8079543-ca0a-44eb-be12-1454fc54a380","Type":"ContainerStarted","Data":"e520f0e9459b1c359bac4ce03f4dd0e30e93e489930f8a5580a7eaf321c00ce8"}
Sep 15 14:15:05.908325 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:05.908296    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:06.617625 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:06.617596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:06.635719 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:06.635694    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"
Sep 15 14:15:06.654482 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:06.654456    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhbx9\" (UniqueName: \"kubernetes.io/projected/d8079543-ca0a-44eb-be12-1454fc54a380-kube-api-access-hhbx9\") pod \"d8079543-ca0a-44eb-be12-1454fc54a380\" (UID: \"d8079543-ca0a-44eb-be12-1454fc54a380\") "
Sep 15 14:15:06.654644 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:06.654596    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8079543-ca0a-44eb-be12-1454fc54a380-maas-api-service-ca\") pod \"d8079543-ca0a-44eb-be12-1454fc54a380\" (UID: \"d8079543-ca0a-44eb-be12-1454fc54a380\") "
Sep 15 14:15:06.655216 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:06.655005    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8079543-ca0a-44eb-be12-1454fc54a380-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8079543-ca0a-44eb-be12-1454fc54a380" (UID: "d8079543-ca0a-44eb-be12-1454fc54a380"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 14:15:06.656989 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:06.656960    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8079543-ca0a-44eb-be12-1454fc54a380-kube-api-access-hhbx9" (OuterVolumeSpecName: "kube-api-access-hhbx9") pod "d8079543-ca0a-44eb-be12-1454fc54a380" (UID: "d8079543-ca0a-44eb-be12-1454fc54a380"). InnerVolumeSpecName "kube-api-access-hhbx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:15:06.756080 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:06.755988    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8079543-ca0a-44eb-be12-1454fc54a380-maas-api-service-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:15:06.756080 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:06.756018    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhbx9\" (UniqueName: \"kubernetes.io/projected/d8079543-ca0a-44eb-be12-1454fc54a380-kube-api-access-hhbx9\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:15:07.485561 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:07.485529    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"
Sep 15 14:15:07.485749 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:07.485558    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx" event={"ID":"d8079543-ca0a-44eb-be12-1454fc54a380","Type":"ContainerDied","Data":"e520f0e9459b1c359bac4ce03f4dd0e30e93e489930f8a5580a7eaf321c00ce8"}
Sep 15 14:15:07.485749 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:07.485613    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e520f0e9459b1c359bac4ce03f4dd0e30e93e489930f8a5580a7eaf321c00ce8"
Sep 15 14:15:07.513368 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:07.513338    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:08.294374 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:08.294337    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:10.962179 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:10.962118    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:11.420234 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:11.420195    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:12.344646 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:12.344611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:13.196367 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:13.196329    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:13.756638 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:13.756605    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:14.608454 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:14.608404    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:15.372813 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:15.372771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:15.476413 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:15.476375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:15.572470 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:15.572437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:15.583277 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:15.583245    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"]
Sep 15 14:15:15.587527 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:15.587502    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-583e9a-kx6bx"]
Sep 15 14:15:15.673235 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:15.673136    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:15.959112 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:15.959019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:16.058711 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:16.058674    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:16.643782 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:16.643747    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8079543-ca0a-44eb-be12-1454fc54a380" path="/var/lib/kubelet/pods/d8079543-ca0a-44eb-be12-1454fc54a380/volumes"
Sep 15 14:15:16.718946 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:16.718912    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:17.565086 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:17.565049    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:18.290484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:18.290450    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:22.177478 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:22.177443    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:22.281899 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:22.281858    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:22.503541 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:22.503455    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:22.559726 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:22.559692    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:15:23.126901 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:23.126820    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:23.235284 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:23.235243    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:24.085647 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:24.085580    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:25.097120 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:25.097086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:25.797996 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:25.797960    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:26.494992 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:26.494960    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:33.537043 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.537006    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"]
Sep 15 14:15:33.537588 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.537250    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42" podUID="3cfbb33b-b1f7-4089-aee2-a6130806c9dd" containerName="maas-api" containerID="cri-o://0c7ccad562e3f5397cc0760a500acf47e0fa06e086c12e0c5fdac7bf1328c401" gracePeriod=30
Sep 15 14:15:33.570568 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.570535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:33.641084 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.641052    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:33.704740 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.704709    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:33.771645 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.771618    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:33.789387 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.789332    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:15:33.833250 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.833218    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:33.930380 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.930345    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-openshift-service-ca\") pod \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") "
Sep 15 14:15:33.930575 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.930502    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p7zv\" (UniqueName: \"kubernetes.io/projected/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-kube-api-access-2p7zv\") pod \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") "
Sep 15 14:15:33.930575 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.930566    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-maas-api-tls\") pod \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") "
Sep 15 14:15:33.930766 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.930598    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-metrics-certs\") pod \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\" (UID: \"3cfbb33b-b1f7-4089-aee2-a6130806c9dd\") "
Sep 15 14:15:33.930822 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.930764    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3cfbb33b-b1f7-4089-aee2-a6130806c9dd" (UID: "3cfbb33b-b1f7-4089-aee2-a6130806c9dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 14:15:33.930935 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.930917    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-openshift-service-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:15:33.933624 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.933583    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-kube-api-access-2p7zv" (OuterVolumeSpecName: "kube-api-access-2p7zv") pod "3cfbb33b-b1f7-4089-aee2-a6130806c9dd" (UID: "3cfbb33b-b1f7-4089-aee2-a6130806c9dd"). InnerVolumeSpecName "kube-api-access-2p7zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:15:33.933764 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.933698    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3cfbb33b-b1f7-4089-aee2-a6130806c9dd" (UID: "3cfbb33b-b1f7-4089-aee2-a6130806c9dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 14:15:33.934156 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:33.934106    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3cfbb33b-b1f7-4089-aee2-a6130806c9dd" (UID: "3cfbb33b-b1f7-4089-aee2-a6130806c9dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 14:15:34.031520 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.031479    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p7zv\" (UniqueName: \"kubernetes.io/projected/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-kube-api-access-2p7zv\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:15:34.031520 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.031512    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-maas-api-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:15:34.031520 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.031521    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfbb33b-b1f7-4089-aee2-a6130806c9dd-metrics-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:15:34.133653 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.133620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:34.326616 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.326586    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:34.636743 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.636710    2582 generic.go:356] "Generic (PLEG): container finished" podID="3cfbb33b-b1f7-4089-aee2-a6130806c9dd" containerID="0c7ccad562e3f5397cc0760a500acf47e0fa06e086c12e0c5fdac7bf1328c401" exitCode=0
Sep 15 14:15:34.637176 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.636829    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"
Sep 15 14:15:34.642797 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.642773    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42" event={"ID":"3cfbb33b-b1f7-4089-aee2-a6130806c9dd","Type":"ContainerDied","Data":"0c7ccad562e3f5397cc0760a500acf47e0fa06e086c12e0c5fdac7bf1328c401"}
Sep 15 14:15:34.642925 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.642815    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42" event={"ID":"3cfbb33b-b1f7-4089-aee2-a6130806c9dd","Type":"ContainerDied","Data":"8b8fb6012c87e977bf11f393a081ba52ab567f43ae679be2fd0134f163ce48d0"}
Sep 15 14:15:34.642925 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.642838    2582 scope.go:117] "RemoveContainer" containerID="0c7ccad562e3f5397cc0760a500acf47e0fa06e086c12e0c5fdac7bf1328c401"
Sep 15 14:15:34.652568 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.652547    2582 scope.go:117] "RemoveContainer" containerID="0c7ccad562e3f5397cc0760a500acf47e0fa06e086c12e0c5fdac7bf1328c401"
Sep 15 14:15:34.652841 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:15:34.652822    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c7ccad562e3f5397cc0760a500acf47e0fa06e086c12e0c5fdac7bf1328c401\": container with ID starting with 0c7ccad562e3f5397cc0760a500acf47e0fa06e086c12e0c5fdac7bf1328c401 not found: ID does not exist" containerID="0c7ccad562e3f5397cc0760a500acf47e0fa06e086c12e0c5fdac7bf1328c401"
Sep 15 14:15:34.652900 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.652851    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7ccad562e3f5397cc0760a500acf47e0fa06e086c12e0c5fdac7bf1328c401"} err="failed to get container status \"0c7ccad562e3f5397cc0760a500acf47e0fa06e086c12e0c5fdac7bf1328c401\": rpc error: code = NotFound desc = could not find container \"0c7ccad562e3f5397cc0760a500acf47e0fa06e086c12e0c5fdac7bf1328c401\": container with ID starting with 0c7ccad562e3f5397cc0760a500acf47e0fa06e086c12e0c5fdac7bf1328c401 not found: ID does not exist"
Sep 15 14:15:34.660320 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.660296    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"]
Sep 15 14:15:34.665472 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:34.665446    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4bb7b3-5c9c96b7cd-x9z42"]
Sep 15 14:15:35.085874 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:35.085844    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:36.049124 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:36.049093    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:36.649664 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:36.649626    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cfbb33b-b1f7-4089-aee2-a6130806c9dd" path="/var/lib/kubelet/pods/3cfbb33b-b1f7-4089-aee2-a6130806c9dd/volumes"
Sep 15 14:15:38.031763 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:38.031728    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:15:40.582442 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:40.582402    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:40.683473 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:40.683439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:15:40.916344 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:15:40.916312    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:17:07.130804 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:17:07.130765    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:17:29.956586 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:17:29.956504    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:18:06.847445 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:18:06.847398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:18:07.898359 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:18:07.898325    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:18:08.656383 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:18:08.656347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:18:30.965893 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:18:30.965854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:18:39.202276 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:18:39.202241    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:19:01.877832 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:19:01.877792    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:19:07.568261 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:19:07.568214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:19:15.878592 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:19:15.878549    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:20:11.128380 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:20:11.128347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:20:11.195149 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:20:11.195110    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:20:16.426301 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:20:16.426210    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:20:16.480701 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:20:16.480656    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:21:17.846445 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:21:17.846412    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:21:22.927935 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:21:22.927901    2582 scope.go:117] "RemoveContainer" containerID="2429e8a20819bd114cdbbed70ee6b3f078d65b8cdc1b42d470ab148be5bc5cda"
Sep 15 14:23:31.706980 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:23:31.706892    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:24:02.592838 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:24:02.592802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:24:08.393788 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:24:08.393734    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:24:16.588416 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:24:16.588379    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:29.444649 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:29.444611    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"]
Sep 15 14:25:29.465161 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:29.465107    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:30.322773 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:30.322739    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"]
Sep 15 14:25:30.349704 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:30.349675    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:30.810552 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:30.810518    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:31.592895 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:31.592861    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:32.404375 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:32.404342    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:33.191157 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:33.191113    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:33.426211 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:33.426176    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:25:40.920013 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:40.919977    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"]
Sep 15 14:25:40.920487 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:40.920196    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx" podUID="4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7" containerName="maas-api" containerID="cri-o://be3a6db9d84512ecf8052d78b9a8c9a97037bd2c12d19883249685709a6718eb" gracePeriod=30
Sep 15 14:25:40.946583 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:40.946556    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:41.008477 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.008443    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:41.108000 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.107964    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:41.164839 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.164805    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:25:41.168707 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.168679    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:41.314737 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.314654    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-openshift-service-ca\") pod \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") "
Sep 15 14:25:41.314907 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.314755    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-maas-api-tls\") pod \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") "
Sep 15 14:25:41.314907 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.314788    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks8h8\" (UniqueName: \"kubernetes.io/projected/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-kube-api-access-ks8h8\") pod \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") "
Sep 15 14:25:41.314907 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.314833    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-metrics-certs\") pod \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\" (UID: \"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7\") "
Sep 15 14:25:41.315114 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.315079    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7" (UID: "4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 14:25:41.315218 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.315190    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-openshift-service-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:41.317198 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.317114    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7" (UID: "4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 14:25:41.317624 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.317600    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7" (UID: "4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 14:25:41.317715 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.317629    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-kube-api-access-ks8h8" (OuterVolumeSpecName: "kube-api-access-ks8h8") pod "4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7" (UID: "4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7"). InnerVolumeSpecName "kube-api-access-ks8h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:25:41.416067 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.416028    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-maas-api-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:41.416067 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.416060    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks8h8\" (UniqueName: \"kubernetes.io/projected/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-kube-api-access-ks8h8\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:41.416067 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.416070    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7-metrics-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:41.561086 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.561046    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:41.650368 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.650331    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:41.822120 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.822082    2582 generic.go:356] "Generic (PLEG): container finished" podID="4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7" containerID="be3a6db9d84512ecf8052d78b9a8c9a97037bd2c12d19883249685709a6718eb" exitCode=0
Sep 15 14:25:41.822312 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.822150    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"
Sep 15 14:25:41.822312 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.822166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx" event={"ID":"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7","Type":"ContainerDied","Data":"be3a6db9d84512ecf8052d78b9a8c9a97037bd2c12d19883249685709a6718eb"}
Sep 15 14:25:41.822312 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.822215    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx" event={"ID":"4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7","Type":"ContainerDied","Data":"5e12ff0c1bbd2fd4e157896f4aa88603125419514f8ae1bee7e07ce5168edd6a"}
Sep 15 14:25:41.822312 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.822236    2582 scope.go:117] "RemoveContainer" containerID="be3a6db9d84512ecf8052d78b9a8c9a97037bd2c12d19883249685709a6718eb"
Sep 15 14:25:41.831980 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.831960    2582 scope.go:117] "RemoveContainer" containerID="be3a6db9d84512ecf8052d78b9a8c9a97037bd2c12d19883249685709a6718eb"
Sep 15 14:25:41.832308 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:25:41.832287    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3a6db9d84512ecf8052d78b9a8c9a97037bd2c12d19883249685709a6718eb\": container with ID starting with be3a6db9d84512ecf8052d78b9a8c9a97037bd2c12d19883249685709a6718eb not found: ID does not exist" containerID="be3a6db9d84512ecf8052d78b9a8c9a97037bd2c12d19883249685709a6718eb"
Sep 15 14:25:41.832365 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.832317    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3a6db9d84512ecf8052d78b9a8c9a97037bd2c12d19883249685709a6718eb"} err="failed to get container status \"be3a6db9d84512ecf8052d78b9a8c9a97037bd2c12d19883249685709a6718eb\": rpc error: code = NotFound desc = could not find container \"be3a6db9d84512ecf8052d78b9a8c9a97037bd2c12d19883249685709a6718eb\": container with ID starting with be3a6db9d84512ecf8052d78b9a8c9a97037bd2c12d19883249685709a6718eb not found: ID does not exist"
Sep 15 14:25:41.843510 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.843484    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"]
Sep 15 14:25:41.850389 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:41.850360    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a090a0-68b8985875-2njkx"]
Sep 15 14:25:42.482616 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:42.482580    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:42.649883 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:42.649844    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7" path="/var/lib/kubelet/pods/4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7/volumes"
Sep 15 14:25:44.119809 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.119769    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:25:44.464501 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.464407    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk" podUID="d95e2950-8361-4453-9464-1c26a365c835" containerName="main" containerID="cri-o://5c6471caa0374751e060409b5c437dafbec120817b3748f3773eb6092548070c" gracePeriod=45
Sep 15 14:25:44.723022 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.722959    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:25:44.840689 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.840654    2582 generic.go:356] "Generic (PLEG): container finished" podID="d95e2950-8361-4453-9464-1c26a365c835" containerID="5c6471caa0374751e060409b5c437dafbec120817b3748f3773eb6092548070c" exitCode=0
Sep 15 14:25:44.840859 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.840698    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"
Sep 15 14:25:44.840859 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.840745    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk" event={"ID":"d95e2950-8361-4453-9464-1c26a365c835","Type":"ContainerDied","Data":"5c6471caa0374751e060409b5c437dafbec120817b3748f3773eb6092548070c"}
Sep 15 14:25:44.840859 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.840784    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk" event={"ID":"d95e2950-8361-4453-9464-1c26a365c835","Type":"ContainerDied","Data":"5bbb82669d4e27c920212c984cbf2ea7227cf535cf3ab0899e86792a750fe5ce"}
Sep 15 14:25:44.840859 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.840806    2582 scope.go:117] "RemoveContainer" containerID="5c6471caa0374751e060409b5c437dafbec120817b3748f3773eb6092548070c"
Sep 15 14:25:44.848091 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.848069    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-dshm\") pod \"d95e2950-8361-4453-9464-1c26a365c835\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") "
Sep 15 14:25:44.848258 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.848129    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47txd\" (UniqueName: \"kubernetes.io/projected/d95e2950-8361-4453-9464-1c26a365c835-kube-api-access-47txd\") pod \"d95e2950-8361-4453-9464-1c26a365c835\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") "
Sep 15 14:25:44.848258 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.848185    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95e2950-8361-4453-9464-1c26a365c835-tls-certs\") pod \"d95e2950-8361-4453-9464-1c26a365c835\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") "
Sep 15 14:25:44.848258 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.848223    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-tmp-dir\") pod \"d95e2950-8361-4453-9464-1c26a365c835\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") "
Sep 15 14:25:44.848471 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.848271    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-home\") pod \"d95e2950-8361-4453-9464-1c26a365c835\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") "
Sep 15 14:25:44.848471 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.848324    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-model-cache\") pod \"d95e2950-8361-4453-9464-1c26a365c835\" (UID: \"d95e2950-8361-4453-9464-1c26a365c835\") "
Sep 15 14:25:44.848578 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.848549    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d95e2950-8361-4453-9464-1c26a365c835" (UID: "d95e2950-8361-4453-9464-1c26a365c835"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:25:44.848785 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.848686    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-home" (OuterVolumeSpecName: "home") pod "d95e2950-8361-4453-9464-1c26a365c835" (UID: "d95e2950-8361-4453-9464-1c26a365c835"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:25:44.848785 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.848764    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-model-cache" (OuterVolumeSpecName: "model-cache") pod "d95e2950-8361-4453-9464-1c26a365c835" (UID: "d95e2950-8361-4453-9464-1c26a365c835"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:25:44.848785 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.848780    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-tmp-dir\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:44.849556 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.849534    2582 scope.go:117] "RemoveContainer" containerID="5c6471caa0374751e060409b5c437dafbec120817b3748f3773eb6092548070c"
Sep 15 14:25:44.849841 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:25:44.849819    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6471caa0374751e060409b5c437dafbec120817b3748f3773eb6092548070c\": container with ID starting with 5c6471caa0374751e060409b5c437dafbec120817b3748f3773eb6092548070c not found: ID does not exist" containerID="5c6471caa0374751e060409b5c437dafbec120817b3748f3773eb6092548070c"
Sep 15 14:25:44.849940 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.849847    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6471caa0374751e060409b5c437dafbec120817b3748f3773eb6092548070c"} err="failed to get container status \"5c6471caa0374751e060409b5c437dafbec120817b3748f3773eb6092548070c\": rpc error: code = NotFound desc = could not find container \"5c6471caa0374751e060409b5c437dafbec120817b3748f3773eb6092548070c\": container with ID starting with 5c6471caa0374751e060409b5c437dafbec120817b3748f3773eb6092548070c not found: ID does not exist"
Sep 15 14:25:44.850627 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.850597    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95e2950-8361-4453-9464-1c26a365c835-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d95e2950-8361-4453-9464-1c26a365c835" (UID: "d95e2950-8361-4453-9464-1c26a365c835"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 14:25:44.850720 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.850684    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95e2950-8361-4453-9464-1c26a365c835-kube-api-access-47txd" (OuterVolumeSpecName: "kube-api-access-47txd") pod "d95e2950-8361-4453-9464-1c26a365c835" (UID: "d95e2950-8361-4453-9464-1c26a365c835"). InnerVolumeSpecName "kube-api-access-47txd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:25:44.850720 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.850695    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-dshm" (OuterVolumeSpecName: "dshm") pod "d95e2950-8361-4453-9464-1c26a365c835" (UID: "d95e2950-8361-4453-9464-1c26a365c835"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:25:44.950321 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.950288    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-model-cache\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:44.950321 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.950318    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-dshm\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:44.950321 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.950327    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47txd\" (UniqueName: \"kubernetes.io/projected/d95e2950-8361-4453-9464-1c26a365c835-kube-api-access-47txd\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:44.950563 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.950337    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95e2950-8361-4453-9464-1c26a365c835-tls-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:44.950563 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:44.950345    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95e2950-8361-4453-9464-1c26a365c835-home\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:45.163724 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.163690    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"]
Sep 15 14:25:45.169995 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.169961    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a090a0/auth-test-model-a090a0-kserve-584596446c-wllsk"]
Sep 15 14:25:45.342772 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.342732    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n" podUID="16edf5d7-b4e1-492f-97e7-48b34ad75cd7" containerName="main" containerID="cri-o://185314415a3726b339e9b0a3c760db989297edb243212923c77469b096c232c0" gracePeriod=45
Sep 15 14:25:45.600478 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.600446    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:25:45.758354 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.758256    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wcx8\" (UniqueName: \"kubernetes.io/projected/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-kube-api-access-8wcx8\") pod \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") "
Sep 15 14:25:45.758354 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.758304    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-tls-certs\") pod \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") "
Sep 15 14:25:45.758354 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.758335    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-model-cache\") pod \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") "
Sep 15 14:25:45.758669 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.758447    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-tmp-dir\") pod \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") "
Sep 15 14:25:45.758669 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.758486    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-home\") pod \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") "
Sep 15 14:25:45.758669 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.758533    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-dshm\") pod \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\" (UID: \"16edf5d7-b4e1-492f-97e7-48b34ad75cd7\") "
Sep 15 14:25:45.758669 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.758617    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-model-cache" (OuterVolumeSpecName: "model-cache") pod "16edf5d7-b4e1-492f-97e7-48b34ad75cd7" (UID: "16edf5d7-b4e1-492f-97e7-48b34ad75cd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:25:45.758875 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.758698    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16edf5d7-b4e1-492f-97e7-48b34ad75cd7" (UID: "16edf5d7-b4e1-492f-97e7-48b34ad75cd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:25:45.758875 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.758775    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-home" (OuterVolumeSpecName: "home") pod "16edf5d7-b4e1-492f-97e7-48b34ad75cd7" (UID: "16edf5d7-b4e1-492f-97e7-48b34ad75cd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:25:45.758980 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.758882    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-model-cache\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:45.758980 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.758897    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-tmp-dir\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:45.758980 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.758907    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-home\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:45.760742 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.760722    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-dshm" (OuterVolumeSpecName: "dshm") pod "16edf5d7-b4e1-492f-97e7-48b34ad75cd7" (UID: "16edf5d7-b4e1-492f-97e7-48b34ad75cd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:25:45.760742 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.760729    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-kube-api-access-8wcx8" (OuterVolumeSpecName: "kube-api-access-8wcx8") pod "16edf5d7-b4e1-492f-97e7-48b34ad75cd7" (UID: "16edf5d7-b4e1-492f-97e7-48b34ad75cd7"). InnerVolumeSpecName "kube-api-access-8wcx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:25:45.760877 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.760733    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16edf5d7-b4e1-492f-97e7-48b34ad75cd7" (UID: "16edf5d7-b4e1-492f-97e7-48b34ad75cd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 14:25:45.848040 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.848004    2582 generic.go:356] "Generic (PLEG): container finished" podID="16edf5d7-b4e1-492f-97e7-48b34ad75cd7" containerID="185314415a3726b339e9b0a3c760db989297edb243212923c77469b096c232c0" exitCode=0
Sep 15 14:25:45.848040 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.848040    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"
Sep 15 14:25:45.848269 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.848073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n" event={"ID":"16edf5d7-b4e1-492f-97e7-48b34ad75cd7","Type":"ContainerDied","Data":"185314415a3726b339e9b0a3c760db989297edb243212923c77469b096c232c0"}
Sep 15 14:25:45.848269 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.848107    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n" event={"ID":"16edf5d7-b4e1-492f-97e7-48b34ad75cd7","Type":"ContainerDied","Data":"06394e3a9ea3c51300b33bd63ab0d7f5f245f0c426c0b567a36b7e1fe96bb56a"}
Sep 15 14:25:45.848269 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.848124    2582 scope.go:117] "RemoveContainer" containerID="185314415a3726b339e9b0a3c760db989297edb243212923c77469b096c232c0"
Sep 15 14:25:45.857540 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.857519    2582 scope.go:117] "RemoveContainer" containerID="185314415a3726b339e9b0a3c760db989297edb243212923c77469b096c232c0"
Sep 15 14:25:45.857839 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:25:45.857819    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"185314415a3726b339e9b0a3c760db989297edb243212923c77469b096c232c0\": container with ID starting with 185314415a3726b339e9b0a3c760db989297edb243212923c77469b096c232c0 not found: ID does not exist" containerID="185314415a3726b339e9b0a3c760db989297edb243212923c77469b096c232c0"
Sep 15 14:25:45.857910 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.857849    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"185314415a3726b339e9b0a3c760db989297edb243212923c77469b096c232c0"} err="failed to get container status \"185314415a3726b339e9b0a3c760db989297edb243212923c77469b096c232c0\": rpc error: code = NotFound desc = could not find container \"185314415a3726b339e9b0a3c760db989297edb243212923c77469b096c232c0\": container with ID starting with 185314415a3726b339e9b0a3c760db989297edb243212923c77469b096c232c0 not found: ID does not exist"
Sep 15 14:25:45.859654 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.859636    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wcx8\" (UniqueName: \"kubernetes.io/projected/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-kube-api-access-8wcx8\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:45.859654 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.859655    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-tls-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:45.859800 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.859665    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16edf5d7-b4e1-492f-97e7-48b34ad75cd7-dshm\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:45.869278 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.869239    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"]
Sep 15 14:25:45.873058 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:45.873033    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d418fd/auth-test-model-d418fd-kserve-b4cd5c447-jfj4n"]
Sep 15 14:25:46.646470 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:46.646441    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16edf5d7-b4e1-492f-97e7-48b34ad75cd7" path="/var/lib/kubelet/pods/16edf5d7-b4e1-492f-97e7-48b34ad75cd7/volumes"
Sep 15 14:25:46.646946 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:46.646832    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d95e2950-8361-4453-9464-1c26a365c835" path="/var/lib/kubelet/pods/d95e2950-8361-4453-9464-1c26a365c835/volumes"
Sep 15 14:25:47.401221 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:47.401190    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:47.508209 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:47.508165    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:47.638910 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:47.638879    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"]
Sep 15 14:25:47.639132 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:47.639100    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerName="istio-proxy" containerID="cri-o://9bcc716de48befbd29a836c51caef6e59149f45d8fb4832c61e9e0459e47bf93" gracePeriod=30
Sep 15 14:25:47.654751 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:47.654663    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:48.106031 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:48.106005    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:48.984491 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:48.984453    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:49.766989 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:49.766940    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:52.199984 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:52.199937    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 14:25:52.200376 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:52.200004    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 14:25:52.891360 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:52.891333    2582 generic.go:356] "Generic (PLEG): container finished" podID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerID="9bcc716de48befbd29a836c51caef6e59149f45d8fb4832c61e9e0459e47bf93" exitCode=0
Sep 15 14:25:52.891492 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:52.891406    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" event={"ID":"21b21433-d06d-4b57-9599-f93b2c09b1b8","Type":"ContainerDied","Data":"9bcc716de48befbd29a836c51caef6e59149f45d8fb4832c61e9e0459e47bf93"}
Sep 15 14:25:52.891492 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:52.891463    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc" event={"ID":"21b21433-d06d-4b57-9599-f93b2c09b1b8","Type":"ContainerDied","Data":"80c707c4e65339af3138334cb92cde1ba093dc624a1b7f4d265687434a7b6489"}
Sep 15 14:25:52.891492 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:52.891479    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80c707c4e65339af3138334cb92cde1ba093dc624a1b7f4d265687434a7b6489"
Sep 15 14:25:52.891601 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:52.891494    2582 scope.go:117] "RemoveContainer" containerID="3e9f6447ebd96af86a9ca9ef5bc77d67966d08bf6a9cb9f46390ab0b179c09fa"
Sep 15 14:25:52.900469 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:25:52.900444    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9f6447ebd96af86a9ca9ef5bc77d67966d08bf6a9cb9f46390ab0b179c09fa\": container with ID starting with 3e9f6447ebd96af86a9ca9ef5bc77d67966d08bf6a9cb9f46390ab0b179c09fa not found: ID does not exist" containerID="3e9f6447ebd96af86a9ca9ef5bc77d67966d08bf6a9cb9f46390ab0b179c09fa"
Sep 15 14:25:52.904002 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:52.903950    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:25:53.031925 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.031887    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-workload-certs\") pod \"21b21433-d06d-4b57-9599-f93b2c09b1b8\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") "
Sep 15 14:25:53.032126 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.031935    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7r8w\" (UniqueName: \"kubernetes.io/projected/21b21433-d06d-4b57-9599-f93b2c09b1b8-kube-api-access-c7r8w\") pod \"21b21433-d06d-4b57-9599-f93b2c09b1b8\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") "
Sep 15 14:25:53.032126 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.031998    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-data\") pod \"21b21433-d06d-4b57-9599-f93b2c09b1b8\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") "
Sep 15 14:25:53.032126 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032038    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-workload-socket\") pod \"21b21433-d06d-4b57-9599-f93b2c09b1b8\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") "
Sep 15 14:25:53.032126 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032083    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-envoy\") pod \"21b21433-d06d-4b57-9599-f93b2c09b1b8\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") "
Sep 15 14:25:53.032389 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032134    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-podinfo\") pod \"21b21433-d06d-4b57-9599-f93b2c09b1b8\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") "
Sep 15 14:25:53.032389 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032186    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-credential-socket\") pod \"21b21433-d06d-4b57-9599-f93b2c09b1b8\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") "
Sep 15 14:25:53.032389 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032212    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-token\") pod \"21b21433-d06d-4b57-9599-f93b2c09b1b8\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") "
Sep 15 14:25:53.032389 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032260    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b21433-d06d-4b57-9599-f93b2c09b1b8-istiod-ca-cert\") pod \"21b21433-d06d-4b57-9599-f93b2c09b1b8\" (UID: \"21b21433-d06d-4b57-9599-f93b2c09b1b8\") "
Sep 15 14:25:53.032389 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032298    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21b21433-d06d-4b57-9599-f93b2c09b1b8" (UID: "21b21433-d06d-4b57-9599-f93b2c09b1b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:25:53.032389 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032341    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "21b21433-d06d-4b57-9599-f93b2c09b1b8" (UID: "21b21433-d06d-4b57-9599-f93b2c09b1b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:25:53.032666 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032451    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21b21433-d06d-4b57-9599-f93b2c09b1b8" (UID: "21b21433-d06d-4b57-9599-f93b2c09b1b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:25:53.032666 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032554    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21b21433-d06d-4b57-9599-f93b2c09b1b8" (UID: "21b21433-d06d-4b57-9599-f93b2c09b1b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:25:53.032786 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032676    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-credential-socket\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:53.032786 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032697    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-workload-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:53.032786 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032710    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-data\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:53.032786 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032723    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-workload-socket\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:53.032936 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.032779    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b21433-d06d-4b57-9599-f93b2c09b1b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21b21433-d06d-4b57-9599-f93b2c09b1b8" (UID: "21b21433-d06d-4b57-9599-f93b2c09b1b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 14:25:53.034368 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.034335    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21b21433-d06d-4b57-9599-f93b2c09b1b8" (UID: "21b21433-d06d-4b57-9599-f93b2c09b1b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 14:25:53.034480 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.034366    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21b21433-d06d-4b57-9599-f93b2c09b1b8" (UID: "21b21433-d06d-4b57-9599-f93b2c09b1b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:25:53.034633 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.034617    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "21b21433-d06d-4b57-9599-f93b2c09b1b8" (UID: "21b21433-d06d-4b57-9599-f93b2c09b1b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:25:53.034679 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.034642    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b21433-d06d-4b57-9599-f93b2c09b1b8-kube-api-access-c7r8w" (OuterVolumeSpecName: "kube-api-access-c7r8w") pod "21b21433-d06d-4b57-9599-f93b2c09b1b8" (UID: "21b21433-d06d-4b57-9599-f93b2c09b1b8"). InnerVolumeSpecName "kube-api-access-c7r8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:25:53.133599 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.133557    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-envoy\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:53.133599 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.133588    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-podinfo\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:53.133599 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.133597    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b21433-d06d-4b57-9599-f93b2c09b1b8-istio-token\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:53.133599 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.133606    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b21433-d06d-4b57-9599-f93b2c09b1b8-istiod-ca-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:53.133862 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.133614    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7r8w\" (UniqueName: \"kubernetes.io/projected/21b21433-d06d-4b57-9599-f93b2c09b1b8-kube-api-access-c7r8w\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:53.893249 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.893197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:53.906076 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.898892    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"
Sep 15 14:25:53.923331 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.923298    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"]
Sep 15 14:25:53.926676 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:53.926650    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a090a0-openshift-default-6676bb465d-r2fjc"]
Sep 15 14:25:54.642810 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:54.642777    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" path="/var/lib/kubelet/pods/21b21433-d06d-4b57-9599-f93b2c09b1b8/volumes"
Sep 15 14:25:58.510385 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.510344    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"]
Sep 15 14:25:58.510851 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.510614    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7" podUID="e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34" containerName="maas-api" containerID="cri-o://b009b2a25c5a175e697ddf2c1a50000e613f212bb33d546f77338839ef6bee1b" gracePeriod=30
Sep 15 14:25:58.540342 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.540312    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:58.615209 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.615166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:58.690533 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.690485    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:58.757470 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.757441    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:58.767064 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.766990    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:25:58.890925 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.890889    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-maas-api-tls\") pod \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") "
Sep 15 14:25:58.891114 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.890967    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j9t7\" (UniqueName: \"kubernetes.io/projected/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-kube-api-access-6j9t7\") pod \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") "
Sep 15 14:25:58.891114 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.891029    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-openshift-service-ca\") pod \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") "
Sep 15 14:25:58.891114 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.891055    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-metrics-certs\") pod \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\" (UID: \"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34\") "
Sep 15 14:25:58.891505 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.891469    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34" (UID: "e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 14:25:58.893107 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.893073    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34" (UID: "e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 14:25:58.893286 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.893269    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-kube-api-access-6j9t7" (OuterVolumeSpecName: "kube-api-access-6j9t7") pod "e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34" (UID: "e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34"). InnerVolumeSpecName "kube-api-access-6j9t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:25:58.893345 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.893277    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34" (UID: "e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 14:25:58.924425 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.924390    2582 generic.go:356] "Generic (PLEG): container finished" podID="e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34" containerID="b009b2a25c5a175e697ddf2c1a50000e613f212bb33d546f77338839ef6bee1b" exitCode=0
Sep 15 14:25:58.924425 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.924432    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"
Sep 15 14:25:58.924683 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.924451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7" event={"ID":"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34","Type":"ContainerDied","Data":"b009b2a25c5a175e697ddf2c1a50000e613f212bb33d546f77338839ef6bee1b"}
Sep 15 14:25:58.924683 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.924492    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7" event={"ID":"e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34","Type":"ContainerDied","Data":"27b554e6b9733a6f691e4e6d398be3016caf14cf85835371beb1f97ca0dcf2cc"}
Sep 15 14:25:58.924683 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.924513    2582 scope.go:117] "RemoveContainer" containerID="b009b2a25c5a175e697ddf2c1a50000e613f212bb33d546f77338839ef6bee1b"
Sep 15 14:25:58.934528 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.934508    2582 scope.go:117] "RemoveContainer" containerID="b009b2a25c5a175e697ddf2c1a50000e613f212bb33d546f77338839ef6bee1b"
Sep 15 14:25:58.934836 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:25:58.934814    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b009b2a25c5a175e697ddf2c1a50000e613f212bb33d546f77338839ef6bee1b\": container with ID starting with b009b2a25c5a175e697ddf2c1a50000e613f212bb33d546f77338839ef6bee1b not found: ID does not exist" containerID="b009b2a25c5a175e697ddf2c1a50000e613f212bb33d546f77338839ef6bee1b"
Sep 15 14:25:58.934900 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.934844    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b009b2a25c5a175e697ddf2c1a50000e613f212bb33d546f77338839ef6bee1b"} err="failed to get container status \"b009b2a25c5a175e697ddf2c1a50000e613f212bb33d546f77338839ef6bee1b\": rpc error: code = NotFound desc = could not find container \"b009b2a25c5a175e697ddf2c1a50000e613f212bb33d546f77338839ef6bee1b\": container with ID starting with b009b2a25c5a175e697ddf2c1a50000e613f212bb33d546f77338839ef6bee1b not found: ID does not exist"
Sep 15 14:25:58.947404 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.947376    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"]
Sep 15 14:25:58.951261 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.951233    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d418fd-6f6d6cc99d-mmms7"]
Sep 15 14:25:58.992765 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.992734    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-openshift-service-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:58.992765 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.992763    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-metrics-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:58.992937 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.992774    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-maas-api-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:58.992937 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:58.992782    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j9t7\" (UniqueName: \"kubernetes.io/projected/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34-kube-api-access-6j9t7\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:25:59.119925 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:59.119893    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:25:59.220910 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:25:59.220879    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:26:00.643667 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:00.643637    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34" path="/var/lib/kubelet/pods/e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34/volumes"
Sep 15 14:26:01.423933 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:01.423895    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-m4784"]
Sep 15 14:26:05.001125 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:05.001088    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:26:05.132473 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:05.132431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:26:05.182056 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:05.182024    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"]
Sep 15 14:26:05.182314 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:05.182267    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" podUID="c8ede612-7215-411e-93d1-625138db97d8" containerName="istio-proxy" containerID="cri-o://8c4b2687164d573a67b24a4fa617cd90a6501efca2fde2e48523018a08c10faf" gracePeriod=30
Sep 15 14:26:05.232768 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:05.232732    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:26:06.115588 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:06.115551    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 14:26:06.115991 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:06.115622    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" podUID="c8ede612-7215-411e-93d1-625138db97d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 14:26:10.438216 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.438187    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:26:10.496210 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496101    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-istio-data\") pod \"c8ede612-7215-411e-93d1-625138db97d8\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") "
Sep 15 14:26:10.496210 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496169    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-istio-envoy\") pod \"c8ede612-7215-411e-93d1-625138db97d8\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") "
Sep 15 14:26:10.496408 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496290    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-workload-socket\") pod \"c8ede612-7215-411e-93d1-625138db97d8\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") "
Sep 15 14:26:10.496408 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496329    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-workload-certs\") pod \"c8ede612-7215-411e-93d1-625138db97d8\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") "
Sep 15 14:26:10.496408 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496366    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ede612-7215-411e-93d1-625138db97d8-istio-token\") pod \"c8ede612-7215-411e-93d1-625138db97d8\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") "
Sep 15 14:26:10.496564 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496412    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ede612-7215-411e-93d1-625138db97d8-istiod-ca-cert\") pod \"c8ede612-7215-411e-93d1-625138db97d8\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") "
Sep 15 14:26:10.496564 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496430    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "c8ede612-7215-411e-93d1-625138db97d8" (UID: "c8ede612-7215-411e-93d1-625138db97d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:26:10.496564 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496461    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ede612-7215-411e-93d1-625138db97d8-istio-podinfo\") pod \"c8ede612-7215-411e-93d1-625138db97d8\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") "
Sep 15 14:26:10.496564 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496498    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-credential-socket\") pod \"c8ede612-7215-411e-93d1-625138db97d8\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") "
Sep 15 14:26:10.496564 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496546    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfzqd\" (UniqueName: \"kubernetes.io/projected/c8ede612-7215-411e-93d1-625138db97d8-kube-api-access-jfzqd\") pod \"c8ede612-7215-411e-93d1-625138db97d8\" (UID: \"c8ede612-7215-411e-93d1-625138db97d8\") "
Sep 15 14:26:10.496808 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496597    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c8ede612-7215-411e-93d1-625138db97d8" (UID: "c8ede612-7215-411e-93d1-625138db97d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:26:10.496808 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496632    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c8ede612-7215-411e-93d1-625138db97d8" (UID: "c8ede612-7215-411e-93d1-625138db97d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:26:10.496808 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496810    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c8ede612-7215-411e-93d1-625138db97d8" (UID: "c8ede612-7215-411e-93d1-625138db97d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:26:10.497021 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496867    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ede612-7215-411e-93d1-625138db97d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c8ede612-7215-411e-93d1-625138db97d8" (UID: "c8ede612-7215-411e-93d1-625138db97d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 14:26:10.497021 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496907    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-credential-socket\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:26:10.497021 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496929    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-istio-data\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:26:10.497021 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496942    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-workload-socket\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:26:10.497021 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.496956    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-workload-certs\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:26:10.498558 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.498535    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ede612-7215-411e-93d1-625138db97d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "c8ede612-7215-411e-93d1-625138db97d8" (UID: "c8ede612-7215-411e-93d1-625138db97d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:26:10.498673 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.498562    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8ede612-7215-411e-93d1-625138db97d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c8ede612-7215-411e-93d1-625138db97d8" (UID: "c8ede612-7215-411e-93d1-625138db97d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 14:26:10.498809 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.498794    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c8ede612-7215-411e-93d1-625138db97d8" (UID: "c8ede612-7215-411e-93d1-625138db97d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 14:26:10.498876 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.498829    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ede612-7215-411e-93d1-625138db97d8-kube-api-access-jfzqd" (OuterVolumeSpecName: "kube-api-access-jfzqd") pod "c8ede612-7215-411e-93d1-625138db97d8" (UID: "c8ede612-7215-411e-93d1-625138db97d8"). InnerVolumeSpecName "kube-api-access-jfzqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:26:10.597632 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.597594    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ede612-7215-411e-93d1-625138db97d8-istio-envoy\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:26:10.597632 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.597624    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ede612-7215-411e-93d1-625138db97d8-istio-token\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:26:10.597632 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.597633    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ede612-7215-411e-93d1-625138db97d8-istiod-ca-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:26:10.597632 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.597641    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ede612-7215-411e-93d1-625138db97d8-istio-podinfo\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:26:10.597900 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.597649    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfzqd\" (UniqueName: \"kubernetes.io/projected/c8ede612-7215-411e-93d1-625138db97d8-kube-api-access-jfzqd\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:26:10.991430 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.991396    2582 generic.go:356] "Generic (PLEG): container finished" podID="c8ede612-7215-411e-93d1-625138db97d8" containerID="8c4b2687164d573a67b24a4fa617cd90a6501efca2fde2e48523018a08c10faf" exitCode=0
Sep 15 14:26:10.991615 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.991455    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"
Sep 15 14:26:10.991615 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.991480    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" event={"ID":"c8ede612-7215-411e-93d1-625138db97d8","Type":"ContainerDied","Data":"8c4b2687164d573a67b24a4fa617cd90a6501efca2fde2e48523018a08c10faf"}
Sep 15 14:26:10.991615 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.991540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g" event={"ID":"c8ede612-7215-411e-93d1-625138db97d8","Type":"ContainerDied","Data":"7acc30338d342b5818cdd4cffda664e78c05119f823043d83e383601ac171a39"}
Sep 15 14:26:10.991615 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:10.991566    2582 scope.go:117] "RemoveContainer" containerID="8c4b2687164d573a67b24a4fa617cd90a6501efca2fde2e48523018a08c10faf"
Sep 15 14:26:11.000634 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:11.000618    2582 scope.go:117] "RemoveContainer" containerID="eb9cb3f1fc4d49d89e1bc3674042bd9d5d8c1cc09666fc29f0bd917c85a57c56"
Sep 15 14:26:11.007413 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:11.007389    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"]
Sep 15 14:26:11.008652 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:11.008632    2582 scope.go:117] "RemoveContainer" containerID="8c4b2687164d573a67b24a4fa617cd90a6501efca2fde2e48523018a08c10faf"
Sep 15 14:26:11.008911 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:26:11.008893    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4b2687164d573a67b24a4fa617cd90a6501efca2fde2e48523018a08c10faf\": container with ID starting with 8c4b2687164d573a67b24a4fa617cd90a6501efca2fde2e48523018a08c10faf not found: ID does not exist" containerID="8c4b2687164d573a67b24a4fa617cd90a6501efca2fde2e48523018a08c10faf"
Sep 15 14:26:11.008966 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:11.008921    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4b2687164d573a67b24a4fa617cd90a6501efca2fde2e48523018a08c10faf"} err="failed to get container status \"8c4b2687164d573a67b24a4fa617cd90a6501efca2fde2e48523018a08c10faf\": rpc error: code = NotFound desc = could not find container \"8c4b2687164d573a67b24a4fa617cd90a6501efca2fde2e48523018a08c10faf\": container with ID starting with 8c4b2687164d573a67b24a4fa617cd90a6501efca2fde2e48523018a08c10faf not found: ID does not exist"
Sep 15 14:26:11.008966 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:11.008939    2582 scope.go:117] "RemoveContainer" containerID="eb9cb3f1fc4d49d89e1bc3674042bd9d5d8c1cc09666fc29f0bd917c85a57c56"
Sep 15 14:26:11.009261 ip-10-0-138-252 kubenswrapper[2582]: E0915 14:26:11.009236    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9cb3f1fc4d49d89e1bc3674042bd9d5d8c1cc09666fc29f0bd917c85a57c56\": container with ID starting with eb9cb3f1fc4d49d89e1bc3674042bd9d5d8c1cc09666fc29f0bd917c85a57c56 not found: ID does not exist" containerID="eb9cb3f1fc4d49d89e1bc3674042bd9d5d8c1cc09666fc29f0bd917c85a57c56"
Sep 15 14:26:11.009343 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:11.009265    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9cb3f1fc4d49d89e1bc3674042bd9d5d8c1cc09666fc29f0bd917c85a57c56"} err="failed to get container status \"eb9cb3f1fc4d49d89e1bc3674042bd9d5d8c1cc09666fc29f0bd917c85a57c56\": rpc error: code = NotFound desc = could not find container \"eb9cb3f1fc4d49d89e1bc3674042bd9d5d8c1cc09666fc29f0bd917c85a57c56\": container with ID starting with eb9cb3f1fc4d49d89e1bc3674042bd9d5d8c1cc09666fc29f0bd917c85a57c56 not found: ID does not exist"
Sep 15 14:26:11.011694 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:11.011675    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d418fd-openshift-default-848d7cf44-tpt4g"]
Sep 15 14:26:12.644551 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:12.644516    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8ede612-7215-411e-93d1-625138db97d8" path="/var/lib/kubelet/pods/c8ede612-7215-411e-93d1-625138db97d8/volumes"
Sep 15 14:26:22.986055 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:26:22.985973    2582 scope.go:117] "RemoveContainer" containerID="9bcc716de48befbd29a836c51caef6e59149f45d8fb4832c61e9e0459e47bf93"
Sep 15 14:27:21.597819 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:27:21.597774    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:27:40.594217 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:27:40.594167    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:27:42.482744 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:27:42.482715    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:27:46.979888 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:27:46.979810    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:30:00.146659 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.146610    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625"]
Sep 15 14:30:00.147205 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147185    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8079543-ca0a-44eb-be12-1454fc54a380" containerName="revoke-keys"
Sep 15 14:30:00.147266 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147208    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8079543-ca0a-44eb-be12-1454fc54a380" containerName="revoke-keys"
Sep 15 14:30:00.147266 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147229    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerName="istio-proxy"
Sep 15 14:30:00.147266 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147238    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerName="istio-proxy"
Sep 15 14:30:00.147266 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147247    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ede612-7215-411e-93d1-625138db97d8" containerName="istio-proxy"
Sep 15 14:30:00.147266 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147256    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ede612-7215-411e-93d1-625138db97d8" containerName="istio-proxy"
Sep 15 14:30:00.147266 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147266    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerName="istio-proxy"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147274    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerName="istio-proxy"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147284    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ede612-7215-411e-93d1-625138db97d8" containerName="istio-proxy"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147293    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ede612-7215-411e-93d1-625138db97d8" containerName="istio-proxy"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147303    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34" containerName="maas-api"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147311    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34" containerName="maas-api"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147323    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95e2950-8361-4453-9464-1c26a365c835" containerName="main"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147332    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95e2950-8361-4453-9464-1c26a365c835" containerName="main"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147344    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7" containerName="maas-api"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147353    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7" containerName="maas-api"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147366    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16edf5d7-b4e1-492f-97e7-48b34ad75cd7" containerName="main"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147374    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="16edf5d7-b4e1-492f-97e7-48b34ad75cd7" containerName="main"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147388    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cfbb33b-b1f7-4089-aee2-a6130806c9dd" containerName="maas-api"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147396    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cfbb33b-b1f7-4089-aee2-a6130806c9dd" containerName="maas-api"
Sep 15 14:30:00.147484 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147488    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cfbb33b-b1f7-4089-aee2-a6130806c9dd" containerName="maas-api"
Sep 15 14:30:00.147955 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147500    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerName="istio-proxy"
Sep 15 14:30:00.147955 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147509    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ede612-7215-411e-93d1-625138db97d8" containerName="istio-proxy"
Sep 15 14:30:00.147955 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147519    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8079543-ca0a-44eb-be12-1454fc54a380" containerName="revoke-keys"
Sep 15 14:30:00.147955 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147529    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cdd651e-5c5c-440e-ba4a-26a2a5c69ac7" containerName="maas-api"
Sep 15 14:30:00.147955 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147538    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95e2950-8361-4453-9464-1c26a365c835" containerName="main"
Sep 15 14:30:00.147955 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147546    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="16edf5d7-b4e1-492f-97e7-48b34ad75cd7" containerName="main"
Sep 15 14:30:00.147955 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.147556    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2eeaf7b-b4b3-4f2b-9b02-d77110b5ed34" containerName="maas-api"
Sep 15 14:30:00.150551 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.150535    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p"]
Sep 15 14:30:00.150677 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.150661    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625"
Sep 15 14:30:00.151089 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.151064    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b21433-d06d-4b57-9599-f93b2c09b1b8" containerName="istio-proxy"
Sep 15 14:30:00.151089 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.151080    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ede612-7215-411e-93d1-625138db97d8" containerName="istio-proxy"
Sep 15 14:30:00.152971 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.152947    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzmkv\""
Sep 15 14:30:00.153796 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.153777    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p"
Sep 15 14:30:00.173230 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.173189    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625"]
Sep 15 14:30:00.175666 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.175636    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p"]
Sep 15 14:30:00.196627 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.196595    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2qq\" (UniqueName: \"kubernetes.io/projected/efa211bd-8d20-47be-b652-df53c9853882-kube-api-access-6d2qq\") pod \"maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625\" (UID: \"efa211bd-8d20-47be-b652-df53c9853882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625"
Sep 15 14:30:00.297329 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.297286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2qq\" (UniqueName: \"kubernetes.io/projected/efa211bd-8d20-47be-b652-df53c9853882-kube-api-access-6d2qq\") pod \"maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625\" (UID: \"efa211bd-8d20-47be-b652-df53c9853882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625"
Sep 15 14:30:00.297502 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.297392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9cjc\" (UniqueName: \"kubernetes.io/projected/26ebb1a3-8276-4e26-ad2b-2ad98b7276c1-kube-api-access-p9cjc\") pod \"maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p\" (UID: \"26ebb1a3-8276-4e26-ad2b-2ad98b7276c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p"
Sep 15 14:30:00.305641 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.305615    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2qq\" (UniqueName: \"kubernetes.io/projected/efa211bd-8d20-47be-b652-df53c9853882-kube-api-access-6d2qq\") pod \"maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625\" (UID: \"efa211bd-8d20-47be-b652-df53c9853882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625"
Sep 15 14:30:00.398031 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.397933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9cjc\" (UniqueName: \"kubernetes.io/projected/26ebb1a3-8276-4e26-ad2b-2ad98b7276c1-kube-api-access-p9cjc\") pod \"maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p\" (UID: \"26ebb1a3-8276-4e26-ad2b-2ad98b7276c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p"
Sep 15 14:30:00.406289 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.406259    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9cjc\" (UniqueName: \"kubernetes.io/projected/26ebb1a3-8276-4e26-ad2b-2ad98b7276c1-kube-api-access-p9cjc\") pod \"maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p\" (UID: \"26ebb1a3-8276-4e26-ad2b-2ad98b7276c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p"
Sep 15 14:30:00.461331 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.461292    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625"
Sep 15 14:30:00.467341 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.467311    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p"
Sep 15 14:30:00.736062 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.736027    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625"]
Sep 15 14:30:00.738133 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:30:00.738106    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa211bd_8d20_47be_b652_df53c9853882.slice/crio-a5b8006b880dca0fea34f8f2b73aa60bb10d98e64f4b6c421c56c3b38ddb8388 WatchSource:0}: Error finding container a5b8006b880dca0fea34f8f2b73aa60bb10d98e64f4b6c421c56c3b38ddb8388: Status 404 returned error can't find the container with id a5b8006b880dca0fea34f8f2b73aa60bb10d98e64f4b6c421c56c3b38ddb8388
Sep 15 14:30:00.741918 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:00.741887    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p"]
Sep 15 14:30:00.743529 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:30:00.743500    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ebb1a3_8276_4e26_ad2b_2ad98b7276c1.slice/crio-97574a34810919ad61ff148ca510ecf562eb3535948ca7cbc4d79e0994ff0ffa WatchSource:0}: Error finding container 97574a34810919ad61ff148ca510ecf562eb3535948ca7cbc4d79e0994ff0ffa: Status 404 returned error can't find the container with id 97574a34810919ad61ff148ca510ecf562eb3535948ca7cbc4d79e0994ff0ffa
Sep 15 14:30:01.131742 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:01.131704    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p" event={"ID":"26ebb1a3-8276-4e26-ad2b-2ad98b7276c1","Type":"ContainerStarted","Data":"3f24abc90d31ecfb928027a8ff8c04dcffc2b4b9f0a13d4b7de4340cd1408dac"}
Sep 15 14:30:01.131934 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:01.131757    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p" event={"ID":"26ebb1a3-8276-4e26-ad2b-2ad98b7276c1","Type":"ContainerStarted","Data":"97574a34810919ad61ff148ca510ecf562eb3535948ca7cbc4d79e0994ff0ffa"}
Sep 15 14:30:01.133340 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:01.133312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625" event={"ID":"efa211bd-8d20-47be-b652-df53c9853882","Type":"ContainerStarted","Data":"2b4e369910c25c930e05e75ced5c71a102ca9fb98ae9c0911b467fcae8cd0149"}
Sep 15 14:30:01.133445 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:01.133356    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625" event={"ID":"efa211bd-8d20-47be-b652-df53c9853882","Type":"ContainerStarted","Data":"a5b8006b880dca0fea34f8f2b73aa60bb10d98e64f4b6c421c56c3b38ddb8388"}
Sep 15 14:30:01.153551 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:01.153503    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p" podStartSLOduration=1.153489113 podStartE2EDuration="1.153489113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:30:01.151910544 +0000 UTC m=+3099.029813364" watchObservedRunningTime="2026-09-15 14:30:01.153489113 +0000 UTC m=+3099.031391922"
Sep 15 14:30:01.167781 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:01.167736    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625" podStartSLOduration=1.167722668 podStartE2EDuration="1.167722668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:30:01.166157232 +0000 UTC m=+3099.044060054" watchObservedRunningTime="2026-09-15 14:30:01.167722668 +0000 UTC m=+3099.045625489"
Sep 15 14:30:02.140591 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:02.140554    2582 generic.go:356] "Generic (PLEG): container finished" podID="26ebb1a3-8276-4e26-ad2b-2ad98b7276c1" containerID="3f24abc90d31ecfb928027a8ff8c04dcffc2b4b9f0a13d4b7de4340cd1408dac" exitCode=0
Sep 15 14:30:02.140782 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:02.140649    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p" event={"ID":"26ebb1a3-8276-4e26-ad2b-2ad98b7276c1","Type":"ContainerDied","Data":"3f24abc90d31ecfb928027a8ff8c04dcffc2b4b9f0a13d4b7de4340cd1408dac"}
Sep 15 14:30:02.142191 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:02.142163    2582 generic.go:356] "Generic (PLEG): container finished" podID="efa211bd-8d20-47be-b652-df53c9853882" containerID="2b4e369910c25c930e05e75ced5c71a102ca9fb98ae9c0911b467fcae8cd0149" exitCode=0
Sep 15 14:30:02.142312 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:02.142238    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625" event={"ID":"efa211bd-8d20-47be-b652-df53c9853882","Type":"ContainerDied","Data":"2b4e369910c25c930e05e75ced5c71a102ca9fb98ae9c0911b467fcae8cd0149"}
Sep 15 14:30:03.296173 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:03.296148    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p"
Sep 15 14:30:03.323922 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:03.323891    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625"
Sep 15 14:30:03.424814 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:03.424723    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d2qq\" (UniqueName: \"kubernetes.io/projected/efa211bd-8d20-47be-b652-df53c9853882-kube-api-access-6d2qq\") pod \"efa211bd-8d20-47be-b652-df53c9853882\" (UID: \"efa211bd-8d20-47be-b652-df53c9853882\") "
Sep 15 14:30:03.424814 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:03.424755    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9cjc\" (UniqueName: \"kubernetes.io/projected/26ebb1a3-8276-4e26-ad2b-2ad98b7276c1-kube-api-access-p9cjc\") pod \"26ebb1a3-8276-4e26-ad2b-2ad98b7276c1\" (UID: \"26ebb1a3-8276-4e26-ad2b-2ad98b7276c1\") "
Sep 15 14:30:03.426831 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:03.426802    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa211bd-8d20-47be-b652-df53c9853882-kube-api-access-6d2qq" (OuterVolumeSpecName: "kube-api-access-6d2qq") pod "efa211bd-8d20-47be-b652-df53c9853882" (UID: "efa211bd-8d20-47be-b652-df53c9853882"). InnerVolumeSpecName "kube-api-access-6d2qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:30:03.426831 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:03.426819    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ebb1a3-8276-4e26-ad2b-2ad98b7276c1-kube-api-access-p9cjc" (OuterVolumeSpecName: "kube-api-access-p9cjc") pod "26ebb1a3-8276-4e26-ad2b-2ad98b7276c1" (UID: "26ebb1a3-8276-4e26-ad2b-2ad98b7276c1"). InnerVolumeSpecName "kube-api-access-p9cjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 14:30:03.526524 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:03.526494    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d2qq\" (UniqueName: \"kubernetes.io/projected/efa211bd-8d20-47be-b652-df53c9853882-kube-api-access-6d2qq\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:30:03.526524 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:03.526521    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9cjc\" (UniqueName: \"kubernetes.io/projected/26ebb1a3-8276-4e26-ad2b-2ad98b7276c1-kube-api-access-p9cjc\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 15 14:30:04.154097 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:04.154066    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p"
Sep 15 14:30:04.154288 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:04.154070    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p" event={"ID":"26ebb1a3-8276-4e26-ad2b-2ad98b7276c1","Type":"ContainerDied","Data":"97574a34810919ad61ff148ca510ecf562eb3535948ca7cbc4d79e0994ff0ffa"}
Sep 15 14:30:04.154288 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:04.154207    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97574a34810919ad61ff148ca510ecf562eb3535948ca7cbc4d79e0994ff0ffa"
Sep 15 14:30:04.155381 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:04.155352    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625" event={"ID":"efa211bd-8d20-47be-b652-df53c9853882","Type":"ContainerDied","Data":"a5b8006b880dca0fea34f8f2b73aa60bb10d98e64f4b6c421c56c3b38ddb8388"}
Sep 15 14:30:04.155506 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:04.155396    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5b8006b880dca0fea34f8f2b73aa60bb10d98e64f4b6c421c56c3b38ddb8388"
Sep 15 14:30:04.155506 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:30:04.155365    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625"
Sep 15 14:34:20.418598 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:20.418569    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-t7n99_e8068551-ad23-4b9d-afe8-2e14a577c031/manager/0.log"
Sep 15 14:34:20.532983 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:20.532953    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-55vgq_04c16753-c34a-4659-81df-a44125a3b449/manager/0.log"
Sep 15 14:34:20.976879 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:20.976845    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79c64c4b4d-4kwl9_b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db/copy-manifests/0.log"
Sep 15 14:34:20.985617 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:20.985592    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79c64c4b4d-4kwl9_b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db/manager/0.log"
Sep 15 14:34:21.844650 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:21.844610    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-b-2aa332-6b8cc76664-4w8cd_fc7fb591-a269-4d8e-8260-d109e79a2708/maas-api/0.log"
Sep 15 14:34:22.599382 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:22.599349    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-a-4441a1-29824695-5g94j_ad343043-b04a-4ea6-958e-fe45cb33503f/cleanup/0.log"
Sep 15 14:34:23.026193 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:23.026096    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-b-2aa332-29824710-2g625_efa211bd-8d20-47be-b652-df53c9853882/cleanup/0.log"
Sep 15 14:34:23.133385 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:23.133358    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w0-1293e7-29824680-qcv52_fa7ce1e0-e84f-451c-930f-1a80e45dcc1f/cleanup/0.log"
Sep 15 14:34:23.235417 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:23.235379    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w0-1293e7-29824695-cjls5_1078ca19-bc0d-4030-8d57-84c5bac3ed4c/cleanup/0.log"
Sep 15 14:34:23.345848 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:23.345819    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w0-1293e7-29824710-bl54p_26ebb1a3-8276-4e26-ad2b-2ad98b7276c1/cleanup/0.log"
Sep 15 14:34:23.450835 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:23.450809    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w4-66261e-29824680-xvtbt_decad930-320d-44ac-99aa-885e98f9815a/cleanup/0.log"
Sep 15 14:34:24.618262 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:24.618233    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-xmn2l_6b412a25-f93d-4c2a-a85e-98487b34ecf0/manager/0.log"
Sep 15 14:34:24.727408 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:24.727377    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-t295p_6259b4e7-a69b-4912-b3dd-f058fcb58cf6/manager/0.log"
Sep 15 14:34:24.827897 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:24.827871    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8lpwx_58376e94-40ae-4172-ab79-58b90943d587/kuadrant-console-plugin/0.log"
Sep 15 14:34:24.943853 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:24.943775    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-2cbx6_d5dd8efd-326c-441c-b845-0a08204e01c3/manager/0.log"
Sep 15 14:34:25.043805 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:25.043778    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-m4784_8c40ccfe-b343-46d8-86d1-840e9af8f4fc/limitador/0.log"
Sep 15 14:34:25.164870 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:25.164840    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-5djd4_9e72947c-265d-4fce-a5ad-4a8341bb14da/manager/0.log"
Sep 15 14:34:25.728331 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:25.728301    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698b9db946-8qhfv_bba8aed7-c12c-4b6f-ae5e-c1445cd935ca/kube-auth-proxy/0.log"
Sep 15 14:34:25.942897 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:25.942864    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-n4f4m_623e2217-eca0-464b-b50b-86e331bf2a7e/istio-proxy/1.log"
Sep 15 14:34:26.707904 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:26.707871    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t_2f29cf7e-3a8e-4d84-803b-accf8fedca0c/payload-processing/0.log"
Sep 15 14:34:27.678646 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:27.678616    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-zhgsk_048f743b-9f78-4129-8c47-e77820f42f68/main/0.log"
Sep 15 14:34:27.780248 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:27.780217    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-st4nc_3d5138aa-cc8f-4b5e-bdae-3e4750a10aaa/main/0.log"
Sep 15 14:34:27.891824 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:27.891795    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvvr54_99775774-70c0-4023-a283-611d584af76b/main/0.log"
Sep 15 14:34:27.993504 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:27.993429    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-tq5tz_20ca03da-589d-49f1-825e-f5b22caf2b26/main/0.log"
Sep 15 14:34:34.138057 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:34.138023    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:34:36.078340 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:36.078307    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:34:36.444768 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:36.444737    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fbj84_b7b844a8-6a0e-4d28-8501-7d96e3cb6dfe/global-pull-secret-syncer/0.log"
Sep 15 14:34:36.486305 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:36.486277    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log"
Sep 15 14:34:36.628350 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:36.628324    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-252.ec2.internal_cd3376ef08c7e4621c42e45696eb6b8e/haproxy/0.log"
Sep 15 14:34:41.345386 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:41.345355    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-xmn2l_6b412a25-f93d-4c2a-a85e-98487b34ecf0/manager/0.log"
Sep 15 14:34:41.379718 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:41.379690    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-t295p_6259b4e7-a69b-4912-b3dd-f058fcb58cf6/manager/0.log"
Sep 15 14:34:41.415707 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:41.415681    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8lpwx_58376e94-40ae-4172-ab79-58b90943d587/kuadrant-console-plugin/0.log"
Sep 15 14:34:41.494788 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:41.494750    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-2cbx6_d5dd8efd-326c-441c-b845-0a08204e01c3/manager/0.log"
Sep 15 14:34:41.515067 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:41.515039    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-m4784_8c40ccfe-b343-46d8-86d1-840e9af8f4fc/limitador/0.log"
Sep 15 14:34:41.558597 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:41.558567    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-5djd4_9e72947c-265d-4fce-a5ad-4a8341bb14da/manager/0.log"
Sep 15 14:34:43.526350 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:43.526318    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6mkh_16a221f0-a870-4a39-8c0c-db1f8c341ea2/node-exporter/0.log"
Sep 15 14:34:43.546008 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:43.545979    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6mkh_16a221f0-a870-4a39-8c0c-db1f8c341ea2/kube-rbac-proxy/0.log"
Sep 15 14:34:43.561855 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:43.561831    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6mkh_16a221f0-a870-4a39-8c0c-db1f8c341ea2/init-textfile/0.log"
Sep 15 14:34:45.645725 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:45.645697    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-kt4ds_4884128c-7457-4add-85ce-103712920e9b/networking-console-plugin/0.log"
Sep 15 14:34:46.257343 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.257316    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gl2rg_519921d5-6953-48a0-92e3-1595172bd6ab/console-operator/0.log"
Sep 15 14:34:46.826328 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.826291    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"]
Sep 15 14:34:46.826684 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.826636    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ebb1a3-8276-4e26-ad2b-2ad98b7276c1" containerName="cleanup"
Sep 15 14:34:46.826684 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.826647    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ebb1a3-8276-4e26-ad2b-2ad98b7276c1" containerName="cleanup"
Sep 15 14:34:46.826684 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.826662    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa211bd-8d20-47be-b652-df53c9853882" containerName="cleanup"
Sep 15 14:34:46.826684 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.826668    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa211bd-8d20-47be-b652-df53c9853882" containerName="cleanup"
Sep 15 14:34:46.826819 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.826732    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa211bd-8d20-47be-b652-df53c9853882" containerName="cleanup"
Sep 15 14:34:46.826819 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.826741    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ebb1a3-8276-4e26-ad2b-2ad98b7276c1" containerName="cleanup"
Sep 15 14:34:46.829822 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.829803    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:46.832172 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.832128    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dnzfz\"/\"kube-root-ca.crt\""
Sep 15 14:34:46.833001 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.832980    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dnzfz\"/\"default-dockercfg-kqx7l\""
Sep 15 14:34:46.833089 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.832988    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dnzfz\"/\"openshift-service-ca.crt\""
Sep 15 14:34:46.838341 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.838316    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"]
Sep 15 14:34:46.984109 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.984076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-podres\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:46.984109 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.984111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-sys\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:46.984344 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.984129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-proc\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:46.984344 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.984277    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-lib-modules\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:46.984344 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:46.984307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk8vp\" (UniqueName: \"kubernetes.io/projected/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-kube-api-access-hk8vp\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:47.085061 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.085028    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-lib-modules\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:47.085248 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.085071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk8vp\" (UniqueName: \"kubernetes.io/projected/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-kube-api-access-hk8vp\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:47.085248 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.085134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-podres\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:47.085248 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.085194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-sys\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:47.085248 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.085215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-proc\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:47.085448 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.085257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-lib-modules\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:47.085448 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.085262    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-podres\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:47.085448 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.085325    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-sys\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:47.085448 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.085334    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-proc\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:47.092339 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.092316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk8vp\" (UniqueName: \"kubernetes.io/projected/c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b-kube-api-access-hk8vp\") pod \"perf-node-gather-daemonset-9z6qz\" (UID: \"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b\") " pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:47.140579 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.140546    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:47.410855 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.410823    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"]
Sep 15 14:34:47.412350 ip-10-0-138-252 kubenswrapper[2582]: W0915 14:34:47.412318    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc8d0c21c_6028_4fa5_becb_cd1a3a4fa58b.slice/crio-f2dbbdfd7b3c6c35b2cf43ae51d615a6577b1f5a4bdc19d9c69f696e90030eff WatchSource:0}: Error finding container f2dbbdfd7b3c6c35b2cf43ae51d615a6577b1f5a4bdc19d9c69f696e90030eff: Status 404 returned error can't find the container with id f2dbbdfd7b3c6c35b2cf43ae51d615a6577b1f5a4bdc19d9c69f696e90030eff
Sep 15 14:34:47.578230 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.578196    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz" event={"ID":"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b","Type":"ContainerStarted","Data":"bef04fe8da649928c0b9be183463d631ca285ee3e6d3f27c0ae6550bd8ded9ba"}
Sep 15 14:34:47.578413 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.578251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz" event={"ID":"c8d0c21c-6028-4fa5-becb-cd1a3a4fa58b","Type":"ContainerStarted","Data":"f2dbbdfd7b3c6c35b2cf43ae51d615a6577b1f5a4bdc19d9c69f696e90030eff"}
Sep 15 14:34:47.578413 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.578368    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:47.593457 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:47.593407    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz" podStartSLOduration=1.593391597 podStartE2EDuration="1.593391597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 14:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 14:34:47.591997157 +0000 UTC m=+3385.469899979" watchObservedRunningTime="2026-09-15 14:34:47.593391597 +0000 UTC m=+3385.471294417"
Sep 15 14:34:48.775747 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:48.775716    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zp7l2_ea8af82a-61dd-44cb-a4a6-27bba6accd04/dns/0.log"
Sep 15 14:34:48.794723 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:48.794697    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zp7l2_ea8af82a-61dd-44cb-a4a6-27bba6accd04/kube-rbac-proxy/0.log"
Sep 15 14:34:48.860660 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:48.860633    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lhd7g_9a1476f2-df89-474d-bbce-2258e2487080/dns-node-resolver/0.log"
Sep 15 14:34:49.420318 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:49.420285    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b7tq6_7edf22ea-7f60-4d71-95db-8594c103b405/node-ca/0.log"
Sep 15 14:34:51.015345 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:51.015238    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698b9db946-8qhfv_bba8aed7-c12c-4b6f-ae5e-c1445cd935ca/kube-auth-proxy/0.log"
Sep 15 14:34:51.065352 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:51.065324    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-n4f4m_623e2217-eca0-464b-b50b-86e331bf2a7e/istio-proxy/0.log"
Sep 15 14:34:51.066601 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:51.066577    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-n4f4m_623e2217-eca0-464b-b50b-86e331bf2a7e/istio-proxy/1.log"
Sep 15 14:34:51.253532 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:51.253504    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t_2f29cf7e-3a8e-4d84-803b-accf8fedca0c/payload-processing/0.log"
Sep 15 14:34:52.293424 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:52.293393    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cqqkh_aaa9a50a-44c0-4d90-9a3b-1855ab51b369/serve-healthcheck-canary/0.log"
Sep 15 14:34:52.861521 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:52.861492    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8mlb_163705e9-255b-49b1-93bf-e0a56c9fa48e/kube-rbac-proxy/0.log"
Sep 15 14:34:52.882004 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:52.881977    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8mlb_163705e9-255b-49b1-93bf-e0a56c9fa48e/exporter/0.log"
Sep 15 14:34:52.918904 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:52.918870    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8mlb_163705e9-255b-49b1-93bf-e0a56c9fa48e/extractor/0.log"
Sep 15 14:34:53.592797 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:53.592770    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dnzfz/perf-node-gather-daemonset-9z6qz"
Sep 15 14:34:55.043849 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:55.043820    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-t7n99_e8068551-ad23-4b9d-afe8-2e14a577c031/manager/0.log"
Sep 15 14:34:55.228486 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:55.228453    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-55vgq_04c16753-c34a-4659-81df-a44125a3b449/manager/0.log"
Sep 15 14:34:55.454936 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:55.454908    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79c64c4b4d-4kwl9_b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db/manager/0.log"
Sep 15 14:34:55.482725 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:55.482694    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79c64c4b4d-4kwl9_b1bc75c5-1cbd-48ba-b0ca-aa52eee9e3db/copy-manifests/0.log"
Sep 15 14:34:57.449411 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:57.449380    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698b9db946-8qhfv_bba8aed7-c12c-4b6f-ae5e-c1445cd935ca/kube-auth-proxy/0.log"
Sep 15 14:34:57.484387 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:57.484358    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-n4f4m_623e2217-eca0-464b-b50b-86e331bf2a7e/istio-proxy/0.log"
Sep 15 14:34:57.486121 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:57.486101    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-n4f4m_623e2217-eca0-464b-b50b-86e331bf2a7e/istio-proxy/1.log"
Sep 15 14:34:57.606094 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:57.606063    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t_2f29cf7e-3a8e-4d84-803b-accf8fedca0c/payload-processing/0.log"
Sep 15 14:34:59.873583 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:59.873541    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698b9db946-8qhfv_bba8aed7-c12c-4b6f-ae5e-c1445cd935ca/kube-auth-proxy/0.log"
Sep 15 14:34:59.909409 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:59.909381    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-n4f4m_623e2217-eca0-464b-b50b-86e331bf2a7e/istio-proxy/0.log"
Sep 15 14:34:59.910089 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:34:59.910071    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-n4f4m_623e2217-eca0-464b-b50b-86e331bf2a7e/istio-proxy/1.log"
Sep 15 14:35:00.040271 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:35:00.040236    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-4441a1-6ff7fbf9cf-bdq5t_2f29cf7e-3a8e-4d84-803b-accf8fedca0c/payload-processing/0.log"
Sep 15 14:35:00.835069 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:35:00.835038    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-92nmn_ac943d06-bf47-4dbf-8d40-8c63e0c48007/konnectivity-agent/0.log" limit=1024
Sep 15 14:35:03.468568 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:35:03.468538    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxgrp_e42c6f90-64b3-4a55-bf82-c9ee9a126dc9/kube-multus-additional-cni-plugins/0.log"
Sep 15 14:35:03.492758 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:35:03.492729    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxgrp_e42c6f90-64b3-4a55-bf82-c9ee9a126dc9/egress-router-binary-copy/0.log"
Sep 15 14:35:03.515875 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:35:03.515850    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxgrp_e42c6f90-64b3-4a55-bf82-c9ee9a126dc9/cni-plugins/0.log"
Sep 15 14:35:03.535645 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:35:03.535619    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxgrp_e42c6f90-64b3-4a55-bf82-c9ee9a126dc9/bond-cni-plugin/0.log"
Sep 15 14:35:03.555543 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:35:03.555513    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxgrp_e42c6f90-64b3-4a55-bf82-c9ee9a126dc9/routeoverride-cni/0.log"
Sep 15 14:35:03.577924 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:35:03.577897    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxgrp_e42c6f90-64b3-4a55-bf82-c9ee9a126dc9/whereabouts-cni-bincopy/0.log"
Sep 15 14:35:03.597603 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:35:03.597578    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxgrp_e42c6f90-64b3-4a55-bf82-c9ee9a126dc9/whereabouts-cni/0.log"
Sep 15 14:35:04.023217 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:35:04.023182    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-s2l72_ae90d50d-e61a-43d9-8286-dc01dc000417/kube-multus/0.log"
Sep 15 14:35:04.137765 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:35:04.137728    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5kxxr_0258dc54-5388-49fa-bd24-0ee885930283/network-metrics-daemon/0.log"
Sep 15 14:35:04.157058 ip-10-0-138-252 kubenswrapper[2582]: I0915 14:35:04.157032    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5kxxr_0258dc54-5388-49fa-bd24-0ee885930283/kube-rbac-proxy/0.log"